Differences

This shows you the differences between two versions of the page.

--- public:radio:2024:yaddnet_data_backup [19/04/24 09:40 BST] – [Backup UDP receive and logrotate] john
+++ public:radio:2024:yaddnet_data_backup [06/03/25 06:49 GMT] (current) – external edit 127.0.0.1
@@ Line 2: / Line 2: @@
-====== Yaddnet Data Backup ======
+====== YaDDNet Data Backup ======
 ** Offsite Text file backups of all DSC messages **
-===== YaDDNet UDP Mirror =====
+  * Alan Spindel initially suggested an offsite backup of the DSC messages in 2019
+  * YaDDNet has had the facility to send copies of each new message out to specified IP addresses via UDP since that time
+  * The DSC messages are sent by UDP as simple plain text, for easy archiving
+  * The archived text file can be used if necessary to rebuild the SQL database table containing the DSC messages in case of disaster
+  * Since I'm now hosting YaDDNet again I decided I should run a UDP log receiver at home to improve the backup provision
+  * The daily archive files will be ''gzipped'' and stored on Dropbox
+    * I could share the link to anyone who is interested in using the saved data
+===== YaDDNet UDP Send to Mirror =====
+  * In the ''PyYadd.py'' function a DSC message is created from the raw incoming data from YaDD (and DSC Decoder) in a standard format.
+  * This involves converting MMSI -> Coast or Ship name, finding the ''[MID]'' and corresponding country etc.
+  * Prior to using the newly created message to inject new data into the SQL server ''PyYadd.py'' calls the function ''send_to_mirror()'' to send out the UDP packet to any configured IP addresses
+  * ''PyYadd.py'' then inserts the new data in the SQL server
+  * Any addressed & listening UDP clients receive the UDP packets and append them to a log file
 <code python>
@@ Line 23: / Line 36: @@
 </code>
 ===== Backup UDP receive and logrotate =====
+==== UDP Listener ====
 <code python>
 #!/usr/bin/env python
+#
+# udp_logger.py
+#
 import SocketServer
 import socket
 import threading
-import re
-#import PyYadd
-#import pysql
-import time
 class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
     def handle(self):
         data = self.request[0].strip()
         socket = self.request[1]
         self.write_file(data)
-        #pysql.load_logger("UDP", "%s : %s" % (data.split(";")[0][1:-1], self.client_address[0]))
     def write_file(self,text):
         filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'
         f = open(filename, 'a+')
-        #timenow = time.strftime("%d/%m/%Y %H:%M:%S", time.gmtime(time.time()))
         log = text
-        #log = " ".join((timenow, text))
-        #print log
         f.write(log+"\n")
         f.close()
         return
 class ThreadedUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer):
@@ Line 69: / Line 70: @@
 if __name__ == "__main__":
-    #pysql.load_logger("yUDPSERV", "Starting YaDD UDP interface....")
     HOST, PORT = "", 2505
     server = ThreadedUDPServer((HOST, PORT), ThreadedUDPRequestHandler)
     ip, port = server.server_address
     server.serve_forever()
     # Start a thread with the server --
-	# that thread will then start one
+    # that thread will then start one
     # more thread for each request
     server_thread = threading.Thread(target=server.serve_forever)
@@ Line 85: / Line 84: @@
 </code>
+  * The UDP Log Receiver listens to UDP/2505 for incoming packets from YaDDNet. These are of the form:
+<code>
+[log];2024-04-19 08:43:06;NLD_MB_HF1;2187.5;SEL;002050480;COAST;205;Belgium;Ostende Radio;SAF;636092799;SHIP;636;Liberia;CISNES, D5HF4, Cargo;TEST;NOINF;--;--;REQ;OK
+[log];2024-04-19 08:43:28;BDXC-Han-Remote;12577.0;SEL;002241022;COAST;224;Spain;Coruna Radio;SAF;538005991;SHIP;538;Marshall Islands;NAVIG8 VIOLETTE, V7IH3, Tanker;TEST;NOINF;--;--;REQ;OK
+</code>
+  * each new packet is appended to the running log file ''/home/gm4slv/yaddlogs/yaddnet_udp_log.txt''
+  * the file will grow indefinitely if left to itself!
+==== Logrotate ===
+  * run by Crontab every day at midnight
+  * copies the current running log to a timestamped file
+  * erases the contents of the running log file
+  * ''gzips'' the timestamped copy and moves it to //Dropbox//
+<code bash>
+## CRONTAB ENTRY
+00 * * * /home/gm4slv/bin/yaddlog_rotate.sh /dev/null 2>&1
+</code>
+<code bash>
+#!/bin/bash
+#
+# yaddlog_rotate.sh
+DROPBOX=/home/gm4slv/Dropbox/Yaddlogs/
+DIR=/home/gm4slv/yaddlogs
+INFILE=yaddnet_udp_log.txt
+TIMESTAMP=`date +%Y%m%d_%H%M`
+OUTFILE=$TIMESTAMP\_$INFILE
+cp $DIR/$INFILE $DIR/$OUTFILE
+cat /dev/null > $DIR/$INFILE
+gzip $DIR/$OUTFILE
+mv $DIR/$OUTFILE.gz $DROPBOX
+</code>
+   * This creates a ''gzipped'' log file in my Dropbox of each day's new DSC messages
+=== Starting the UDP Listener at boot-up ===
+  * crontab entry to start it as a background job at ''@reboot''
+<code bash>
+#
+# CRONTAB ENTRY
+#
+@reboot python /home/gm4slv/bin/udp_logger.py &
+</code>
 --- //John Pumford-Green 19/04/24 09:31 BST//
+Last updated : ~~LASTMOD~~
 ===== Further Information =====
+[[https://www.yaddnet.org/index.php | YaDDNet]]
-{{tag>}}
+{{tag>radio yaddnet}}