Our full technical support staff does not monitor this forum. If you need assistance from a member of our staff, please submit your question from the Ask a Question page.


Log in or register to post/reply in the forum.

Telnet and reformatting data


40324Weather Sep 15, 2014 05:55 PM

First off, I want to thank anyone for their assistance. I have a CR1000 with my sensor setup at a 05103 wind/dir, HMP35C thermometer and humidity, TB4 Rain Gauge, PTB101 pressure (CS105), and Li-Cor 200x. now right now for the sake of programming below I currently do not have the wind or pyrometer in play, so all I did was take the code out of the Wunderground upload string, and when I add those feature I will just modify the code to add that to the data string, and I am comfortable with that.

now you will see some stuff I added as well to calculate a running 15 min, 1 hour, 3 hour, 6 hour, 12 and 24 hour rainfall. I have a huge request to have that available. I am hoping one day I can produce that into a simple html file and have it upload that to a FTP server I have access to, but that may be for another post. I think I can tackle that.

ON TO MY GOAL AND DILEMMA

Let me first show you my current code that works great thus far...


'CR1000
'Created by Short Cut (3.0)
'
'/////////////////////////////////////////////////////////////////////////////////////////
'/////////////////////////// V A R I A B L E S & U N I T S /////////////////////////////
'/////////////////////////////////////////////////////////////////////////////////////////
'
Dim AirTC
Public AirTF
Public RH
Public BP_inHg
Public WS_mph
Public WindDir
Public SlrkW
Public Rain_midnight
Public BattV
Public PTemp_C
Public SlrMJ
Public WC_F
Public Tdf
Public RainMeasured
Public RainSum_24hr
Public RainSum_12hr
Public RainSum_6hr
Public RainSum_3hr
Public RainSum_1hr
Public RainSum_15min
Public RainAverage_24hr
Public RainAverage_12hr
Public RainAverage_6hr
Public RainAverage_3hr
Public RainAverage_1hr
Public RainAverage_15min
Public Count24
Public Count12
Public Count6
Public Count3
Public Count1
Public Count15
Public PWSGetRequest As String * 600
Public PWSGetResponse As String * 500
Public Socket
'UTC TIme Variables
Dim TimeLong As Long
Public UTCTime As String * 30
Const UTC_OFFSET = -4*3600 '-6 Hours GMT (Adjust for your time zone)
Dim UTCTime$ As String * 30
Public UTCTime$$ As String * 30
'Units
Units BattV = Volts
Units PTemp_C = Deg C
Units BP_inHg = inHg
Units RainMeasured = inch
Units RainAverage_24hr = inch
Units RainAverage_12hr = inch
Units RainAverage_6hr = inch
Units RainAverage_3hr = inch
Units RainAverage_1hr = inch
Units RainAverage_15min = inch
Units AirTF = Deg F
Units RH = %
Units WS_mph = miles/hour
Units WindDir = degrees
Units WC_F = Deg F
Units Tdf = Deg F
Units SlrkW= kW/m^2
Units SlrMJ= MJ/m^2
'
'/////////////////////////////////////////////////////////////////////////////////////////
'//////////////////////////////// D A T A T A B L E S ///////////////////////////////////
'/////////////////////////////////////////////////////////////////////////////////////////
'
DataTable(TimeTable,true,1)
TableHide
Sample(1,TimeLong,NSEC)
EndTable
'
DataTable(Min5,True,-1)
DataInterval(0,5,Min,10)
Average(1,BP_inHg,FP2,False)
Totalize(1,RainMeasured,FP2,False)
Average(1,AirTF,FP2,False)
Maximum(1,AirTF,FP2,False,True)
Minimum(1,AirTF,FP2,False,True)
Sample(1,RH,FP2)
Average(1,WS_mph,FP2,False)
Maximum(1,WS_mph,FP2,False,True)
Minimum(1,WS_mph,FP2,False,True)
Sample(1,WindDir,FP2)
Average(1,SlrkW,FP2,False)
Totalize(1,SlrMJ,IEEE4,False)
EndTable
'
DataTable(Daily,True,-1)
DataInterval(0,1440,Min,10)
Minimum(1,BattV,FP2,False,False)
Average(1,BP_inHg,FP2,False)
Totalize(1,RainMeasured,FP2,False)
Average(1,AirTF,FP2,False)
Maximum(1,AirTF,FP2,False,True)
Minimum(1,AirTF,FP2,False,True)
Sample(1,RH,FP2)
Average(1,WS_mph,FP2,False)
Maximum(1,WS_mph,FP2,False,True)
Minimum(1,WS_mph,FP2,False,True)
Sample(1,WindDir,FP2)
EndTable

'Main Program
'
'/////////////////////////////////////////////////////////////////////////////////////////
'/////////////////////////////// M A I N P R O G R A M //////////////////////////////////
'/////////////////////////////////////////////////////////////////////////////////////////

BeginProg
'Main Scan
Scan(3,Sec,1,0)
'Default Datalogger Battery Voltage measurement 'BattV'
Battery(BattV)
'Default Wiring Panel Temperature measurement 'PTemp_C'
PanelTemp(PTemp_C,_60Hz)
'CS105 Barometric Pressure Sensor measurement 'BP_inHg'
PortSet(2,1)
VoltSe(BP_inHg,1,mV2500,3,1,0,_60Hz,0.184,626.339)
BP_inHg=(BP_inHg*0.02953)+0.18
'TB4 Rain Gauge measurement 'Rain_in'
PulseCount(RainMeasured,1,1,2,0,0.01,0)
'running totals updated each scan'
AvgRun(RainAverage_24hr,1,RainMeasured,28800)
AvgRun(RainAverage_12hr,1,RainMeasured,14400)
AvgRun(RainAverage_6hr,1,RainMeasured,7200)
AvgRun(RainAverage_3hr,1,RainMeasured,3600)
AvgRun(RainAverage_1hr,1,RainMeasured,1200)
AvgRun(RainAverage_15min,1,RainMeasured,300)
If Count24<28800 Then Count24=Count24+1
If Count12<14400 Then Count12=Count12+1
If Count6<7200 Then Count6=Count6+1
If Count3<3600 Then Count3=Count3+1
If Count1<1200 Then Count1=Count1+1
If Count15<300 Then Count15=Count15+1
RainSum_24hr=Count24*RainAverage_24hr
RainSum_12hr=Count12*RainAverage_12hr
RainSum_6hr=Count6*RainAverage_6hr
RainSum_3hr=Count3*RainAverage_3hr
RainSum_1hr=Count1*RainAverage_1hr
RainSum_15min=Count15*RainAverage_15min
'rainfall since midnight calculation'
Rain_midnight=Rain_midnight+RainMeasured
If IfTime(0,1440,Min) Then Rain_midnight=0
'HC2S3 (constant power) Temperature & Relative Humidity Sensor measurements 'AirTF' and 'RH'
Therm107(AirTF,1,1,1,0,_60Hz,1.8,32)
PortSet(1,1)
Delay(0,150,mSec)
VoltSe(RH,1,mV2500,2,0,0,_60Hz,0.1,0)
PortSet(1,0)
RH=RH+1.1
If RH>100 AND RH<108 Then RH=100
'05103 Wind Speed & Direction Sensor measurements 'WS_mph' and 'WindDir'
PulseCount(WS_mph,1,2,1,1,0.2192,0)
BrHalf(WindDir,1,mV2500,4,2,1,2500,True,0,_60Hz,355,0)
If WindDir>=360 Then WindDir=0
If WindDir<=-.00001 Then WindDir=NAN
'Wind Chill calculation 'WC_F'
WC_F=35.74+0.6215*AirTF-35.75*WS_mph^0.16+0.4275*AirTF*WS_mph^0.16
If WC_F>AirTF OR WC_F=NAN Then WC_F=AirTF
If AirTF>50 OR WS_mph<3 Then WC_F=AirTF
'Solar radiation measurement'

'Dew Point calculation 'TdF'
AirTC=(5/9)*(AirTF-32)
DewPoint(Tdf,AirTC,RH)
If Tdf>AirTC OR Tdf=NAN Then Tdf=AirTC
Tdf=1.8*Tdf+32
'Call Data Tables and Store Data
CallTable(Min5)
CallTable(Daily)
NextScan
'
SlowSequence
Scan(5,min,0,0)
'Track current time
'Calculate UTC Time for Wunderground Output
TimeLong = Public.TimeStamp(1,1) - UTC_OFFSET
CallTable(TimeTable)
UTCTime = TimeTable.TimeLong(4,4) 'yyyy/MM/dd hr:mm:ss
UTCTime$ = Replace (UTCTime," ","+"):UTCTime$$ = Replace (UTCTime$,":","%3A")
'Refer to http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol for PWS upload protocol documentation
'Change ID (yourid) and PASSWORD (yourpassword) to specific site settings
PWSGetRequest="GET /weatherstation/updateweatherstation.php?ID=KKYGEORG1&PASSWORD=Sundays442&dateutc="+UTCTime$$ +"&humidity="+FormatFloat(RH,"%0.2f")+"&rainin="+FormatFloat(RainSum_1hr,"%0.2f")+"&dailyrainin="+FormatFloat(Rain_midnight,"%0.2f")+"&tempf=" _
+FormatFloat(AirTF,"%0.2f")+"&baromin="+FormatFloat(BP_inHg,"%0.2f")+"&solarradiation="+FormatFloat(SlrkW,"%0.2f")+"&dewptf="+FormatFloat(Tdf,"%0.2f")+"&action=updateraw"+CHR(13)+CHR(10)
'ADD THESE LATER "&solarradiation="+FormatFloat(SlrkW,"%0.2f")+
Socket=TCPOpen("rtupdate.wunderground.com",80,1024)
If Socket<>0 Then
'Transmit GET request
SerialOut(Socket,PWSGetRequest,"",0,0)
'Based on tests a legitimate User-Agent request-header field is required
SerialOut(Socket,"User-Agent: Mozilla/5.0"+CHR(13)+CHR(10),"",0,0)
'Based on tests a Host request-header field is not required
SerialOut(Socket,"Host: rtupdate.wunderground.com"+CHR(13)+CHR(10),"",0,0)
'Based on tests a final blank line (carriage return and line feed) is required
SerialOut(Socket,CHR(13)+CHR(10),"",0,0)
'Capture the GET request response for debugging purposes
SerialIn(PWSGetResponse,Socket,500,"",200)
EndIf
NextScan
EndSequence
EndProg
'
'/////////////////////////////////////////////////////////////////////////////////////////
'//////////////////////////////////////// E N D //////////////////////////////////////////
'/////////////////////////////////////////////////////////////////////////////////////////EndProg

Now I think what I am needing is another slowscan, I would prefer it to happen every 10 minutes but maybe not at exactly 10 minutes 20 minutes 30 minutes 40 minutes after the hour, I would like to offset it like 9, 19, cause I guess the servers get hit hard on 5 and 10 minute intervals...

CW0003>APRS,TCPXX*:@241505z4220.45N/07128.59W_032/005g008t054r001p078
P048h50b10245e1w

is an example they used, for example my station without the wind would appear as

AP106>APRS,TCPXX*:@241505z4220.45N/07128.59W_.../...g...t054r001p078P048h50b10245e1w

information I found on where the telnet has to go to

1. CWOP data are sent in real-time from your Personal Weather Station (PWS) via the
Internet using a TELNET connection (not FTP!) or through a Ham Radio wireless broadcast to a site with an APRS IS connection (I-gate)
2. Data are forwarded to an Automatic Position Reporting System (APRS)
"Tier 2" server, typically "arizona.aprs2.net" port 23, which is the primary access point for
weather messages in APRS

also their is comment on alternate servers

arizona.aprs2.net port 23
socal.aprs2.net port 23
indiana.aprs2.net port 23
newengland.aprs2.net port 23
aprsfl.net port 14580

now I have read the time stamp @241505z cane omitted case the server times it when its received and doesn't look at the stamp, so that doesn't require the weird 2 digits for the day of the month, and UTC time.

CW0003>APRS,TCPXX*:@241505z4220.45N/07128.59W_032/005g008t054r001p078
P048h50b10245e1w

How is the weather data coded into the data packet?

When you look at examples of APRS position weather packets here, or here, the part after the longitude "E" or "W" carries the weather data as symbols followed by numbers. The underscore "_" followed by 3 numbers represents wind direction in degrees from true north. This is the direction that the wind is blowing from. The slash "/" followed by 3 numbers represents the average wind speed in miles per hour. The letter "g" followed by 3 numbers represents the peak instaneous value of wind in miles per hour. The letter "t" followed by 3 characters (numbers and minus sign) represents the temperature in degrees F. The letter "r" followed by 3 numbers represents the amount of rain in hundredths of inches that fell the past hour. The letter "p" followed by 3 numbers represents the amount of rain in hundredths of inches that fell in the past 24 hours. Only these two precipitation values are accepted by MADIS. The letter "P" followed by 3 numbers represents the amount of rain in hundredths of inches that fell since local midnight. The letter "b" followed by 5 numbers represents the barometric pressure in tenths of a millibar. The letter "h" followed by 2 numbers represents the relative humidity in percent, where "h00" implies 100% RH. The first four fields (wind direction, wind speed, temperature and gust) are required, in that order, and if a particular measurement is not present, the three numbers should be replaced by "..." to indicate no data available. Solar radiation data can also be coded into the data packet.

the e1w at the end is the type of station this can be any 4 digit character it says, and I think for campbell scientific station we can use CSci

If anyone can help me with this, I can yes it on my system, Im just trying to see example code where people are rounding data and then adding leading zeros, to make it 3 digits, and more so is how to write code to do the telnet.

What I am hoping is once I see some code I can build the data fields like I learned from the Wunderground. and in the public field I will see the data that is being sent so I can troubleshoot.

Thanks for all your help and I will try to answer any questions, I know this will help others, I just would not find any example code for these systems for CWOP and its all about Wunderground which I tackled and learned a lot from.


Ron Malinowski


GaryTRoberts Sep 15, 2014 08:04 PM

Ron,

I can give you a hand with this as I am doing it here on one of my test stations in the yard here (see AG1T-11 and AG1T-3). I use HTTPPost to put my data in the CWOP system instead of Telnet http://www.aprs-is.net/SendOnlyPorts.aspx. It is a lot easier. I have a subroutine that sends the APRS data. Here is a copy of my code that will have to be modified for your program. You can see the results at http://aprs.fi/weather/a/AG1T-11 when my test station is up and running. I am a licensed radio amateur, which makes getting the username and password needed to access the network easier, but the concept is the same for normal CWOP stations.

Sub SUBSendAPRSData()
IPRoute("sixth.aprs.net", 4)

' AG1T-11 Station (West Yard Test Station)
aprs_http_header = "Accept: */*" & CHR(10)
aprs_http_header = aprs_http_header & "Accept-Type: text/plain" & CHR(10)
aprs_http_header = aprs_http_header & "Content-Type: application/octet-stream"

aprs_message = "user AG1T pass 1234 vers CRBasicAPRS 2013111200" & CHR(10)
temp_number = wind_direction
Sprintf(temp_string, "%03.0f", temp_number)
aprs_message = aprs_message & "AG1T-11>APECSI:!4145.95N/11151.26W_" & temp_string
temp_number = wind_speed_mph
Sprintf(temp_string, "%03.0f", temp_number)
aprs_message = aprs_message & "/" & temp_string & "g..."
aprs_message = aprs_message & "t" & FormatLong(air_temperature_f, "%03u") & "r...p...P..."
aprs_message = aprs_message & "L" & FormatLong(slrw, "%03u")
aprs_message = aprs_message & "h" & FormatLong(relative_humidity, "%02u")
temp_number = pressure
temp_number = (temp_number * 10)
Sprintf(temp_string, "%05.0f", temp_number)
aprs_message = aprs_message & "b" & temp_string
aprs_message = aprs_message & "xCSIL" & PROGRAM_VERSION 'WinAPRS with "CSI L"ogger designation I made up
aprs_message = aprs_message & CHR(10)
aprs_message = aprs_message & CHR(10)

aprs_http_post_socket = HTTPPost(APRS_URL & CHR(58) & APRS_PORT, aprs_message, aprs_http_response, aprs_http_header)

TCPClose(aprs_http_post_socket)
EndSub '//End SubSendAPRSData

That might be a bit to get your head around. Feel free to shoot me an email at gtroberts -at- campbellsci.com and I will be glad to give you a hand getting the data in the CWOP system.

* Last updated by: GaryTRoberts on 9/15/2014 @ 2:53 PM *


GaryTRoberts Sep 15, 2014 08:14 PM

I also call my APRS subroutine in a slowsequence so it does not mess up my measurements (and other vital stuff).

SlowSequence
Scan(10, Min, 0, 0)
'Send data to the APRS network. Do this first as the xively data is done over https and takes the logger some to process
' the https certificates.
Call SUBSendAPRSData()

NextScan
EndSequence

The subroutine can be wrapped up in a If(IfTime()) to make it happen at odd intervals if needed as well.


GaryTRoberts Sep 15, 2014 08:18 PM

Also, here is some formatting help that might be useful as well from an email I received earlier today helping another customer:

You can format a number with leading zeros using sprintf() / formatfloat() / formatlong().

For example instead of

If Month <10 Then NMonth = "0"+Month Else NMonth = Month

you could use

NMonth = FormatLong(Month,"%02u")

or

NMonth = FormatFloat(Month,"%02.0f")


Using SprintF() you can also so some inline concatenation that can be handy for dates

For example,

Public DateTime As String * 32
Public rTime(9) As Long
Alias rTime(1) = Year,Month,DayOfMonth,Hour,Minute,Second,microSecond,WeekDay,DayofYear


BeginProg
Scan (3,Sec,0,0)
RealTime (rTime)
'create string formatted as YYYY-MM-DD_hh-mm-ss
Sprintf (DateTime,"%04u-%02u-%02u_%02u-%02u-%02u",Year,Month,DayOfMonth,Hour,Minute,Second)
NextScan
EndProg

Looking at the help for these instructions in CRBasic will give you better details than I can here.


40324Weather Sep 16, 2014 02:42 AM

Gary,

Thank you for your response, I have printed out your responses and I will start to build some code into mine and Ill see how far I can get with it, I think you will get me 80% on first try Im hoping and appreciate your willingness to help.

Will be in touch, my personal email is wx4gps@me.com

I only have 20days of CRBasic knowledge, :-) Appreciate you helping me reach my goals

Ron

Log in or register to post/reply in the forum.