Network Traffic Analysis With DD-WRT Netflow and Ntop

Introduction

DD-WRT include the capability of running rflow, a Cisco Netflow data exporter implementation. The netflow data is sent to a port of a computer (management server) on your LAN running a Netflow collector, in this case this is ntop.
Ntop is an open source network traffic monitoring tool that shows the network usage via a web browser. The web interface for monitoring, configuration and administration make ntop easy to use and suitable for monitoring various kind of networks. 

Router Setup

My router is running the DD-WRT v24-sp2 firmware (you need a version with rflow support).
Rflow can monitor the available interfaces of the router, in my case these are br0 (Lan & Wlan), vlan0, eth1, Wlan0 and WAN.
Log into your router through your browser, than go to the Services | Services where you will find the RFlow / MACupd section.
RFlow: select "Enable"
Server IP: The IP address of your computer that wil run ntop.
This computer must have a static IP address, or using a DHCP static lease.
Port: The UDP port that will be used to send the netflow information.
Common default ports for Netflow are 2055 and 9996. In my first setup I used port 2055 but because this port was used by other applications I had to change to 9996 to make my installation stable.
Interface: LAN&WLAN
Then click "Apply Settings"

Setup ntop management server

Ntop will run on many operating systems. The ntop Web site offers multiple versions (sources and binaries) of the package for download. The free binary version for Windows is limited to capturing only the first 2000 packets. This limitation does not exist in the Linux versions and the paid Windows version. I use the free unlimited version ntop for win32 v3.2 from OPENXTRA. You can find the easy to install OPENXTRA version on several places on the Internet like here [1]
To run ntop for win32 I use an old Windows XP laptop (320MB RAM) as terminal-, web-, file-, print- and scanner server in my network and this laptop proofs to have enough resources to do this additional job.
After the installation is finished, you should have a new icon in the system tray called OPENXTRA Commander. Double-click this icon to open the OPENXTRA Commander. If the NTop Service plug-in is not started, click Start in the Action column to start it. Once it is started, click the Launch action for the NTop plug-in, which will open your browser (http://localhost:3000). If all is well, you will already be collecting some impressive data. In the Windows Control panel | Administrative Tools | Services you can check if the ntop service is running. Optional you can configure the service to restart automatic after a failure.

Rfow configuration

You have to create a virtual rflow interface. Do this by selecting Plugins | All in the menu listing at the top of the webpage.
In the Active column click on "NO" next to NetFlow to enable the plugin.
Click on "NetFlow" in the Configure column.
Click on "Add NetFlow Device".
NetFlow Device Name: Any name you like, I choose DD-WRT.
Click on "Set Interface Name".
Local UDP Collector Port: Use the same port as configured in the router (I used 9996).
Click on "Set Port".
Virtual Netflow Interface Network Address: Your LAN network address and its netmask.
If your router is using 192.168.1.1, then this should be: 192.168.1.0/255.255.255.0. Ntop uses this address to recognize the local hosts from the remote hosts.
Click on "Set Interface".
Leave everything else now to defaults.
Now we have two interfaces ntop can monitor, the NIC of the local computer and the Netflow interface. We now can switch which interface we wish to monitor.
In the menu at the top select Admin | Switch NIC.
Under Available Network Interface select the NetFlow Device name you entered earlier (DD-WRT in my case).
Then click on Switch to NIC.
You will notice that some menu entries have a small padlock icon in them; these are the Web pages that require a password to access. The default credentials for the XTRA package is user = admin, password = admin.
You can take a first view of the web pages of ntop.

Rrd configuration

ntop stores all of its active data in RAM, so if the system is reset, you lose all your data. Archiving of this data to disk must be configured. Ntop is using rrd to prevent that your disk fills up. RRD uses a round robin database which stores time-series data in a very compact way so that it will not expand over time. You have to configure what data with what detail must be archived and with what retention time.
Do this by selecting Plugins | All in the menu listing at the top of the webpage.
In the Active column click on "NO" next to rrdPlugin to enable the plugin.
Then click on "rrdPlugin" in the Configure column to show the RRD Preferences.
I let the first 6 items (interval to delay) unchanged.
Data to dump: Hosts and Interfaces
Hosts Filter: I choose to only archive data of local hosts with my LAN address 192.168.1.0/255.255.255.0
RRD detail: I choose medium
RRD Files Path: This is the location where rrd stores his data: \NTopWin32\rrd.

Ports

Ntop will tell you on different pages which ports are used. In the Windows version you will find the file "services" (without extension) in the directory C:\Program Files\OPENXTRA\NTopWin32.
The structure of the file is

port         port         alias        # description
name         number/tcp  

like this:

pop2         109/tcp   pop-2        # Post Office Protocol - V2
pop3         110/tcp   pop-3        # Post Office Protocol - Version 3
sunrpc       111/udp   rpcbind

You can add portnames to this file if you want ntop to show a portname 

in stead of a port number. I have added the follwing ports:

#
# Extra services
#
ssh          22/tcp
ms-sql-s     1433/tcp
ms-sql-s     1433/udp
ms-sql-m     1434/tcp
ms-sql-m     1434/udp
upnp         1780/tcp
upnp         1780/udp
ssdp         1900/udp
ntop         3000/tcp
rdp          3389/tcp
rdp          3389/udp
atq          3456/udp
nat-t        4500/udp
remotescan   6077/tcp
remotescan   6078/udp
netflow      9996/udp
dropbox      17500/tcp
dropbox      17500/udp

Stop and start the ntop service for the changes to take effect.

Protocols

There is a default list of protocols ntop will monitor for you, if you want you can define a smaller list of protocols or you can add protocols to the list. To do this you have to create a protocol.list in the directory C:\Program Files\OPENXTRA\NTopWin32. The ntop web page will display the protocols in the same order as they are defined in the list, you can change the order if you like.
The structure of the file is protocolname=port, where port is a portname that you can find in the file services or a portnumber.
Behind the = you can define multiple ports with port|port| or with 12-20.
In the following file I have first defined the default values and added some extra protocols.

## Default ntop protocollist ##
FTP=ftp|ftp-data|69
HTTP=http|www|https|3128
DNS=name|domain
Telnet=telnet|login
NBios-IP=netbios-ns|netbios-dgm|netbios-ssn
Mail=pop-2|pop-3|pop3|kpop|smtp|imap|imap2
DHCP-BOOTP=67-68
SNMP=snmp|snmp-trap
NNTP=nntp
NFS/AFS=mount|pcnfs|bwnfs|nfsd|nfs|nfsd-status|7000-7009
VoIP=5060|2000|54045
X11=6000-6010
SSH=22

## Default ntop Peer-to-Peer protocols ##
Gnutella=6346|6347|6348
Kazaa=1214
WinMX=6699|7730
DC++=-1
eDonkey=4661-4665
BitTorrent=6881-6999|6969

## Default ntop Messenger protocols ##
Messenger=1503|1863|5000|5001|5190-5193

## Extra ntop protocols ##
Comodo=1037-1045|1280|4447-4448
Avast=1281-1282
NetFlow=9996
UPnP=1780|1900
Dropbox=17500
Ntop=3000
Remotescan=6077|6078
RDP=3389
Streaming=554|1755|1935|3689|4070|5222|7070
Nat-t=4500
IIS=1025|3456
SQL=1433
LDAP=ldap|ldaps
RPC=111
SLP=427
LPR=515|631

After you have created the file you have to configure ntop to use this file. Select in the menu listing at the top of the webpage Admin | Configure | Startup Options and select the IP Preferences link. In the field TCP/UDP Protocols To Monitor (-p) specify C:\Program Files\OPENXTRA\NTopWin32\protocols.list (must be the full path). See ntop internal help for more information. Stop and start the ntop service for the changes to take effect.

Usage

In the Ntop Bandwidth Monitoring Guide you can find some interesting usage scenario's, like:

• Who are the top internet bandwidth users on my network?
• What websites do the top bandwidth users visit?
• What websites get the most traffic from within my organization?
• What websites' traffic consumes most of my bandwidth?
• What applications are being used?
• Which local hosts share the most data?
• At what time of the day is the network most Utilized?
• Performing a network inventory
• Exporting traffic data
• Detecting network security violations?

Suppose you identify a particular host as the major consumer of bandwidth, what if you want to find out just what exactly he is doing online that is consuming so much bandwidth? Here is how ntop can help:

1. Identify the host you are interested in [one way is to sort on the Data for Network Traffic stats for local hosts.
2. Click on that host to bring up the Info about xxxxx page where xxxx is the name or IP address of the host you are interested in.
3. Scroll down to the bottom of the page to the Active TCP/UDP Sessions table. A screen is shown which "lays it all out for you".

Host Fingerprints
You can switch the interface you want to monitor. You should remember when you use the NetFlow interface that NetFlow does not send you the actual packages like the local NIC interface. This is why ntop cannot report fingerprints when the Netflow interface is used. You can find Host Fingerprints in the menu IP | Local | Host Fingerprints.

Local Matrix
In the menu IP | Local | Local Matrix you will, when using the NetFlow interface, see no traffic between local hosts. This is caused by the default behaviour of a switch that is used to connect your local hosts to the router. Only the traffice between local hosts and remote hosts is captured.

Historical data
Historical data can be viewed with ntop (or other tools) by clicking on the icon on webpages like Info about Host and Plugin | Round Robin Databases | Arbitrary Graphs.

Dumping Ntop Data
There are scripts to dump data in a MySQL database on sourceforge.net. However, within ntop, just click Utils|Data Dump to show a dialog box. You can dump data about different objects into different formats – see the ntop guide for the formats. Some of these formats are importable into a spreedsheet and from there you can unleash the full power of Open Office Calc or Excel unto your traffic data.\

Workarounds

In this version of ntop there are a few small annoyances to work around.

Hostname resolving
PC's in my network are not always on, I found when ntop starts he resolves the hostnames of al host in the network, but after some time when these PC's switch off and on ntop forget the hostname. For me it was no problem to restart the ntop service with a bat file every night with a scheduled task of Windows. The bat file look like this

@echo off
REM - File: Daily Restart ntop for win32.bat
REM - Description: Restart ntop for Win32 Service tbv name resolving
REM - Author: Jan S
echo Restarting ntop for Win32...
echo ======================================================
net stop "ntop for Win32"
net start "ntop for Win32"
echo ======================================================
echo ntop for Win32 Restarted

Broken links
On several pages links are used to services on the Internet for additional information. There are links to WHOIS information and there is a link voor ASN information.
There are several ways to solve this problem, because I use Firefox the easiest way for me was url rewriting in the browser, you can use the Firefox Add-On Redirector to do that.

WHOIS was http://www.radb.net/cgi-bin/radb/whois.cgi?obj=*
can be rewritten to http://whois.domaintools.com/$1 or http://www.lookip.net/$1 or any other you like.

ASN was http://ws.arin.net/cgi-bin/whois.pl?queryinput=*
ca be rewritten to https://apps.db.ripe.net/search/query.htm?searchtext=$1
FAQ, in the menu About there is a link to FAQ but the faq.html file is not local available. You can copy the file from www.ntopsupport.com/faq.html to C:\OPENXTRA\NTopWin32\html\faq.html.
There is a lot of important information available here.

Finally

The combination of a DD-WRT router running rflow and a PC running ntop provides a low cost solution for remote network traffic usage and activity (NetFlow monitoring). Rflow provides fast packet capture and also captures packets efficiently thus preserving CPU cycles. With ntop Luca Deri has created a brilliant tool for seeing what is happening on your network in realtime. This is only a basic tutorial to show what you can do with DD-WRT and rflow. ntop has many more possibilities, out of the box via configuration and via extra scripts which are available in the directory C:\OPENXTRA\NTopWin32\www and on the Internet.
There is a new version 5 available for Windows with many more possibilities, it's worth looking at it.

External Links

• Configuring Ntop to work with the DD-WRT firmware for the Linksys WRT54G(S) router
• Configuring ntop
• Ntop Netflow with a WRT54GS Firewall/Router and NST Probe
• RFlow Collector and MySQL
• Ntop, persistent data and rrd
• Ntop Bandwidth Monitoring Guide
• Ntop Users Guide
• Ntop Tutorial Slides

This information is published earlier by me at DD-WRT.com

Disable NetBios, enable DNS with DD-WRT

Introduction

Using a LAN router based on DD-WRT software offers extra features and functionality sets to improve your network, easy and for low costs. In this article I explain the setup of DNS/DHCP for standard name resolution while disabling NetBIOS.

NetBIOS

NetBIOS is a legacy API from the early days (1983) of PC networking providing services for applications on seperate PC's to communicate in a single local PC network. NetBIOS uses therefore a flat namespace of NetBIOS names and makes extensive use of broadcasting for name resolving of adresses.
SMB is an application-layer network protocol mainly used for file- and print sharing. SMB can run on top of the Session (and lower) network layers in several ways:

• Via the NetBIOS API, which in turn can run on several transports:
• On several legacy protocols such as NBF, NBX and Pathworks. 
• On UDP ports 137, 138 & TCP ports 137, 139 (NBT);
• Directly over TCP, port 445 (Direct hosting of SMB over TCP/IP).

The first implementations of NetBIOS run directly on the Link Layer by the NBF protocol, later NetBIOS came available on routable networks to support existing NetBIOS based applications.

The support of legacy services to support old applications has resulted in complexity and limitations because of the use multiple Naming Systems (NetBIOS name and Hostname, NetBIOS Node types, WINS and DNS, LMHOSTS and HOSTS, .. ) and multiple network management utilities. Communication on Internet requires a hierarchical namespace and worldwide standards for addressing and network management. Because new applications do not depend on NetBIOS anymore, disabling NetBIOS and using the same standards is the way to go for:

• simplification, improvement security and speed,
• removing multiple methods of nameresolution,
• standardizing on TCP/IP name resolution on DNS for file and printer sharing.  

Impact of disabling NetBIOS
Shutting off NetBIOS reduces the network's browsing functionality because the Computer Browser - the service manifested in Network Neighborhood, My Network  Places, and the Net View command - sits atop NetBIOS. Connecting to a share doesn't change: 

• net view command to view resources of a host with hostname
• net use command to access a share
• using UNC name in Start, Run or in address bar of Internet browser
• connect a drive from My Computer or Windows Explorer
• add a Shortcut to your My Network Places folder

In our case users has predefined access to resources, so limited browsing functionality is not an issue, as network manager there are plenty of free tools available for browsing the network.
Disabling NetBIOS means that the infamous ports (UDP 137 and 138, TCP 137 and 139) are not used anymore, direct hosting of SMB over TCP/IP uses UDP en TCP port 445, you might have to update your firewalls for these changes.

Disable NetBIOS
To disable NBT manually, change the computer's TCP/IP properties. In the Network Connections window, you'll see an object for each network card on your system. Right-click the network card for which you want to disable NBT, then choose Properties. On the Properties page, double-click the Internet Protocol (TCP/IP) object, then click Advanced on the Internet Protocol (TCP/IP) Properties page. Click the WINS tab, then click the Disable NetBIOS over TCP/IP radio button. Clear the Enable LMHOSTS lookup check box, then click OK until you've closed the pages.
To verify that you've killed NBT, you can type "ipconfig /all" on a command line. You'll see a line confirming that NetBT is disabled. You can centrally disable NBT with DHCP, I will that explain later.

Disable and remove unnecessary services with care
The Computer Browser is of no use anymore. Type services.msc at the command prompt and select the Computer Browser to stop and disable the service.

The service TCP/IP NetBIOS Helper Service should, despite the name not be disabled. If you stop this service you will receive a system error 1231 with the message "The network location cannot be reached". On various websites it is erroneously recommended to stop this service. In fact it is a legacy name from the time SMB was related to NetBIOS, according to Bill Grant a better name would be TCP/IP SMB Helper.

In My Network Places I disabled Entire Network with the registry key "NoEntireNetwork".

DNS 

The Internet maintains two principal namespaces, the domain namespace and the IP address namespace. DNS maintains the domain namespace and provides translation services between it and the address namespace.
Unlike the flat namespace of Netbios, DNS has a hierarchical namespace, organized in subordinate levels (subdomains) of the DNS root domain. A hostname in DNS is a fully qualified domain name (FQDN), this is a name that is completely specified in the hierarchy of the DNS, having no parts omitted. The dot in a DNS name is required to force a DNS lookup.
The IP address namespace is logically recognized as consisting of two parts: the network prefix and the host identifier. The subnet mask or the CIDR prefix determines how the IP address is divided into network and host parts.

The DNS hostname of a Windows computer is based on the Computername and the DNS suffix: <comptername>.<dns suffix>., like barebone.example.com with example.com as suffix.
The Computername is set during the installations of Windows as a Computer property, the DNS suffix will be centrally managened with dnsmasq to get DNS hostnames for all devices, including mobile devices such as phones and tablets. When the computername is used the system will automatically append the DNS suffix.

Best practice for internal domain name
A domain name must be carefully planned, the best practice for an internal domain name is to use a sub-domain of an external registered domain. Only if you are sure not using an external domain in the future you could use for example .lan (not .local) as TLD that is what I use in the example configuration.

Dnsmasq

Introduction
Dnsmasq is a lightweight DNS and DHCP server available at the DD-WRT LAN Router. Dnsmasq accepts DNS queries and either answers them or forwards them to a real, recursive, DNS server. It also answers DNS queries for DHCP configured hosts.
The dnsmasq DHCP server supports dynamic and static address assignments and multiple networks. It automatically sends a sensible default set of DHCP options, and can be configured to send any desired set of DHCP options, including vendor-encapsulated options.

Our local network with the subnets 192.168.2.0/24 and 192.168.3.0/24 is shown in the figure, the router configuration for the two subnets are explained in a previous blog.

Prerequisites
All our network devices must have their own computername, and be configured to get their network configuration via DHCP. If you have some machines with static IP addresses DNSmasq will incorporate them as well based on the hardware (MAC) address, so you don't need to change them.

Configuration

De configuration of dnsmasq in DD-WRT consists of two parts: a Basic Setup and an Additional Setup at the Tab Services.
Let's start with the Tab Basic Setup in the webinterface of the DD-WRT router.
On this page we enable DNS and DHCP for the subnet 192.168.3.0 with dynamic IP adresses.
Select Use DNSMasq for DHCP, Use DNSMasq for DNS and DHCP-Authorative.

Under the TAB Services we select Services for the remaining settings.
Under DHCP Server choose LAN&WLAN, this is your internal domain, the Domain Name at the Basic Setup page is the external domain.
Under LAN Domain we definine our internal domain name, in our case its is lan.
Under static leases you define the MAC Addresses, the computername and static IP Address.
Under DNSMasq we enable both DNSMasq and Local DNS.

The remaining settings must be defined in Additional DNSMasq Options. The syntaxt of the settings you can find in the DNSMasq manual. My network have these settings:

dhcp-range=wan,192.168.2.0,static                          Enables DHCP for subnet 192.168.2.0 with static IP Addresses (for dynamic addresses you can define the start and end address). 
wan is the label of this subnet to reference to in the following dhcp-options.
By default dnsmasq sends some standard options to DHCP clients:
- netmask and broadcast address are set to the same as the host running dnsmasq,
- the DNS server and the default Gateway are set to the address of the machine running dnsmasq,
- if the domain option is set, it is send as connection specific suffix and DNS suffix search list.
With dhcp-option these values can be overridden and additional options set. 
dhcp-option=wan,3,192.168.2.1                               Default Gateway (3) as the ISP router 
dhcp-option=wan,6,192.168.2.3                               DNS Server (6)

ptr-record=3.2.168.192.in-addr.arpa,DD-WRT.lan    Return a PTR DNS record for the Lan router.
local=/lan/                                                              Queries from .lan are not send to upstream server.
dhcp-option=43,01:04:00:00:00:02                           Disable NetBIOS over TCP/IP, this requires that the DHCP client is configured Default in Advanced TCP/IP settings.
This DHCP option must be defined last for Windows XP client, see Microsoft KB953761.

Final check
You can check your configuration in the file dnsmasq.conf by entering a cat /tmp/dnsmasq.conf in the Command Shell at the Tab Administration Command.
With WinSCP you can manage your configuration in a user friendly explorer like interface.
With ipconfig /all you can check if the configuration of your clients are correctly set, special attention for the DNS suffix and the message that NetBIOS over TCP/IP is disabled.

That's it: I think dnsmasq is easy to use but powerfull software.

External Links

• NetBIOS on Wikipedia
• Server Message Block (SMB)
• Direct hosting of SMB over TCP/IP
• Live Without NetBIOS 
• Network Location Cannot be Reached
• DNS on Wikipedia
• DHCP Architecture
• Dnsmasq Manual

Setup an extra Router in your LAN

Introduction 

In our local network I use a Linksys WRT54GL as LAN Router behind the ADSL Modem/Firewall/Gateway of the Internet Service Provider (ISP). On the LAN router I have installed DD-WRT router firmware, this is free Open Source software. The rationale to add the LAN Router is using the extra features and functionality set to

• enhance security, speed and understandability
• modernize and standardize with easy central management at low cost 

In this article I explain the configuration of the routers to create transparant communication between hosts at both sites of the router. In later blogs I will explain the configuration of other functions. Our local network is shown in the figure below:

The server provides terminalservices, file services, print services, scan services and management services. The server is based on Windows XP SP3 with only 320 MB memory, the clients are based on Windows XP, Vista and Windows 7, the mobiles are Android Tablets and Phones and iPhones. The Wireless AccessPoint is configured as a bridge at a central point in the building. I still use Windows XP with limited resources to force myself to efficiently configure the server.

IP Routing

Hosts and networks
IP adressing is based on the concept of hosts and networks. A host is essentially anything on the network that is capable of receiving and transmitting IP packets on the network, such as a PC, a Server or a Router.
The hosts are connected together by one or more networks. An IP address is 32 bits wide, and as said, it is composed of two parts: the network number, and the host number (like zip code and house number).
The subnet mask or the CIDR prefix determines how the IP address is divided into network and host parts. By convention, it is expressed as four decimal numbers separated by periods, such as 192.168.2.1.
In our case the network number consists of the first three numbers (192.168.x), this is defined with the netmask 255.255.255.0 or with the 192.168.x.0/24 CDIR notation where 24 is the number of bits of the network number. In our local network we use two (sub)networks with the IDs: 192.168.2.0 and 192.168.3.0. The host numbers per network can range from 1 to 254, 255 is a broadcast address.

Use of ARP
Each host has a hardware (or MAC) address, which is six bytes long, this is a unique identifier assigned to network interfaces. They are normally written in hexadecimal form separated by dashes (02-FE-87-4A-8C-A9 for example).
Suppose the laptop on the network 192.168.3.0 wants to send a packet to the server for the first time, and it knows the IP address of the server. To send a packet, the laptop needs to know the hardware address of the server. The Address Resolution Protocol (ARP) is used for dynamic discovery of this address. ARP caches IP addresses and corresponding hardware adresses he has discovered. ARP wil broadcast a request when the IP address is not yet in the cache.
In the case the laptop on the network 192.168.2.0 wants to send a packet to the server on the 192.168.3.0 network the Router must forward the packet from the 192.168.2.0 network to the 192.168.3.0 network. This is accomplished by having the laptop use the hardware address of the Router and the IP address of the server. The Router will forward the packet to the hardware address of the server. These hardware addresses are obtained using ARP as described earlier. Hosts that cannot be reached on the same network must be forwarded by a Router.

Routing
In the IP configuration of each host the IP address of a Default Gateway is defined, this Gateway is the router that forwards packets to other networks.
In the case of the 192.168.2.0 network there are two Routers, the LAN Router will forward the packets to the 192.168.3.0 network and the ISP Router will forward the packets for other networks on the Internet. This means that on the 192.168.2.0 network, besides the Default Gateway to the Internet, the Route to the 192.168.3.0 network must me defined.

Router Configuration

Configuration of the ISP Router
The route from the 192.168.2.0 network to the 192.168.3.0 network must be configured as a Static Route in the Router of the ISP with [Destination IP=192.168.3.0, Gateway=IP address LAN router].

When the laptop in the 192.168.2.0 network wants to send a packet to the server in the 192.168.3.0 network, the laptop sends it to the Default Gateway, this is the Router of the ISP, this Router forward the packet to the LAN Router, who will forward the packet to the server.
When the laptop on the 192.168.3.0 network sends a packet to a host on the Internet, the laptop sends the packet first to his Default Gateway, this is the LAN Router, who forwards the packet of his Default Gateway, this is the ISP Router, who forwards the packet to the host on the Internet. When a packet returns from the Internet, the ISP router knows the Route to the server through the Gateway defined in the Static Route.

Configuration of the LAN Router
The WAN port of the LAN router must be connect to a LAN port of the ISP router.
Most commodity routers has two Operating Modes, Gateway or Router. By default the router is configured with the operating mode Gateway. Gateway mode is intended for the connection to the Internet, Gateway mode gives you firewall and network address transalation (NAT) functionality which makes the LAN addresses invisible on the WAN side. The ISP Router must keep the default Gateway operating mode setting but for the LAN router the operating mode must be changed to Router to make network addresses visible from both sides of the router. You should also want to disable the firewall on the LAN router to allow full communication between the local networks.

As was described earlier the Default Gateway of the LAN router is the ISP Router.
With these changes we are able to communicate to devices throughout the network and have added new features and functionality.

Correcting distortions in scanned maps

In a previous blog I wrote about some challenges using historical maps in OziExplorer.

Distortions
Another challenge can be image distortions by scanning old paper maps. There are several causes for distortions such as scanner mechanical, paper deformation and humidity. When you view the scanned image, sometimes you can see that lines that should be straight, are not. If you calibrate (georeference) the map in OziExplorer and display the lat/lon grid, you will see that the grid of the image does not match with the grid displayed by OziExplorer. These distortions are not linear in the map, in some parts the difference is for example +1 mm, in some other -5 mm or so.

Resample
A georeference definies the transformation between the pixel row/colum coordinate in the image and the X/Y map coordinate in the real world.
About the grid points in the map we know both coordinates, when we are able to define the relation between them better, with a non-linear transformation, it is possible to resample the image to get a map that can be used better in OziExplorer, representing correct lat/lon positions of any pixel of the image. The resample software can calculate the correct new pixel coordinates of the grid points and transform the closed-by pixels according to them in the new image.

On the Internet you can find several image processing software products for correcting distortions. In this blog I write about my experiences how this can be done with Ilwis.

Ilwis
Ilwis (Integrated Land and Water Information System) is open source GIS & Remote Sensing software, developed by ITC up to release 3.3 in 2005. ILWIS comprises a complete package of image processing, spatial analysis and digital mapping.
Because it has rich functionality you need some time to learn using it, but there is a very good user guide with extensive tutorials and it has a full on-line help.

Download the software and the tutorials and do some exercises to introduce yourself to ILWIS. When you are ready to correct your scanned images take the follwing steps:

Step 1 Get your image in the right file format
If necessary convert your input file to Tif format (it can be done with IrfanView).

Step 2 Import your image in Ilwis
In the main window of Ilwis select Open>Import>Ilwis>Raster>Tif.
Select the input file
Choose a name for the output file
and click OK and a Ilwis raster map is produced.

Step 3 Create a Georeference for the raster map
A Georeference is the same as a callibration in OziExplorer. In this step we define the controlpoints and the non-linear transformation to resample the image.
In the Map Window select File>Create Georeference.
Choose a Georeference name (I choose the same name as the raster map).
Give it some Description
Select the radio button Georeference Tiepoints.
Coordinate system> push the button to create one.
Choose a Coordinate system name (I choose Nord de Guerre).
Give a description (I choose: Coordinate system for the M831 map serie).
Select the radiobutton CoordSystem Projection
I used the 1SP parameters with the scale factor, if the 2SP parameters are used the scale factor is 1.
Projection: Lambert Conformal Conic
Ellipsoid: Du Plessis Modified
False Easting: 600000
False Northing: 300000
Central Meridiaan: 07 44' 13.95"
Central Parallel: 49 30' 00"
Scale Factor: 0.999509081
Standard Parallel 1: 49 30'00"
Standard Parallel 2: 49 30'00"
Datum: User defined, dx= 1383, dy=44, dz=454
Press OK

Step 4 Digitize the grid points (Tie points)
With the opened Georeference Editor you can digitize the grid points with your mouse. In the dialog box you see the local pixel coordinates, you have to type in the XY values from the map. It depends on the distortions in the map how many points you need, I took many.
Choose a non-linear transformation method, like Full Second Order, look to the Sigma value (standard deviation), the lower the value the better the results.
To check the results you can display grid lines and see how they match with the grid lines of the map.
Stop the editor and select in the map window: Layers>Add Grid Lines
Choose as Grid distance: 1000 m
Choose as color: white
Press OK
You can remove the grid lines with Remove layer in the map window.
Add/change tiepoints and/or change the transformation method with the Georeference Editor as the results are not good enough.

In the Map window you can check the georeference by selecting File>Open Pixel Information. In the Pixel Information window select: File>Add Coordinate system> LatLon and LatLonWGS84. When you move your curser in the map windows you can check the XY and LatLon coordinates in the WGS84 and NDG datum.

Step 5 Resample map
In the Operation List, select Resample
Choose the input rastermap
Choose the resample method
Choose a name for the resampled output file
Choose the target georeference: New, this is a Georeference Corner
Give the pixel size: I choose 2.5 (it is a 1: 25000 map)
Press Ok and your map is being resampled
When it is ready you have to export the resampled raster map to GeoTif, you can do that with File>Export.

Step 6 Load the corrected map in OziExplorer
The GeoTiff file can directly be loaded in OziExplorer,it uses the information in the header of the file to use the callibration points.
In OziExplorer select File>Import Map>Single DRG Map
OziExplorer guides you in the import proces.









The left picture is the uncorrected map, the distortions are in different places in different directions, the right picture is the corrected. When you enlarge the picture you can see the corrections.

I think Ilwis is a great software package.

Using Historical Maps in OziExplorer

Using historical maps in OziExplorer can be a challenge because of the many standards used in the world of cartography software. Searching the internet, testing and help from experts (Louisiana State University, University of Twente, Kadaster Netherlands and the OziExplorer user forum) was needed, in this blog I publish my findings to
1: Convert projection parameters and
2: Calculate missing datum shift parameters.

In the Map Collection section of the Library of Congress I found a nice serie of Dutch 1:25000 topographic maps, called M831. These maps are made by the US Army Map Service during World War II. You can download them in Jpeg2000 format, because this is not a widely supported format you have to convert them to a more common format. With the freeware program Irfanview you can open these files and save in another format.

Digital Maps
A digital map is an image which has been georeferenced (calibrated) so OziExplorer can use any pixel position on the map to determine the true geographic position. To do this calibration you need data about the projection and the datum of the map. The projection and the datum for this mapserie are both not supported by OziExplorer. When searching the internet I discovered that I had to understand where I was searching for.

Map Projection
A map projection is a means of projecting the spherical earth onto a flat plane. On the border of the maps we find information of the projection used:
Projection:   Lambert Conical Orthomorphic
Origin:         49° 30'N and 7° 44'13.95"E
Scale factor: 0.999509081

OziExplorer does not support this projection but mathematically it is synonymous with the Lambert Conformal Conic, which is supported. Orthomorphic is a classical British term. The "British definition" defines this projection at the Origin with one standard parallel (1SP) and a scale factor where the "American definition" defines this projection with two standard parallels (2SP). From the one standard parallel, the scale factor and the ellipsoidal flattening (explained later at map datum) it is possible to derive the equivalent two standard parallels and then treat the projection as a two standard parallel Lambert Conformal Conic. On the internet I could not find conversion software, geo experts told that they don't know of such software. It is possible to determine an estimate by trial and error with software like PCTrans, I preferred to write a Javascript (see 1SPto2SP at the bottom of the page) to calculate the values. The formulas I have used are from table 2 of the document "A new Slovak mapping projection compatible with GPS".
The 2 standard parallels for our mapserie are then:
First standard parallel    : 47° 41' 28.11"N
Second standard parallel: 51° 17' 11.74"N

Calibration
In OziExplorer we can now select Load and Calibrate Map Image in the File menu. We setup the projection with the calculated projection parameters and georeference (calibrate) the map by using the grid points of the lat/lon grid on the map as calibration points. The setup of the Map Datum comes later. In the same setup tab we use "Show/hide Corner Markers" to place the Corner Markers on the border around the actual map (called the neat line). This is necessary so OziExplorer can detect where the actual map boundaries are to change maps when you cross this boundary when using moving map real time tracking mode or for drawing grids till the border or when you use MapMerge for OziExplorer utility. After saving the calibration in the .map file you can check the quality by displaying the lat/lon grid with Grid Line Setup.

Map Datum
In the previous step we have georeferenced the map with the local lat/lon grid on the map. The lat/lon coordinates of places on the map depend on what mathematical definition is used to represent the earth when the map was drawn. This reference is called the Datum. The shape of the earth is best defined as a ellipsoid, the defining parameters of the ellipsoid are the semi-major axis (equatorial radius) length (a) and the semi-minor axis ( polar radius) length (b). Mostly it is defined with a and 1/f, where f=(a-b)/a, f is the ellipsoidal flattening. At least eight constants are needed to form a complete datum:
* 3 to specify the location of the origin,
* 3 to specify the orientation of the lat/lon coordinate system and
* 2 to specify the dimensions of the ellipsoid.
In the early days of geodesy, the datum could only be determined for a relatively small area from a local origin on the surface of the earth. This has lead to a proliferation of different ellipsoids with a different size and shape and with other origins and orientations. This is why every country has his own local Map Datum. For example, the maps in the Netherlands are based on the Bessel ellipsoid of 1841 with Amersfoort as origin.

On the border of our maps we find information about the map datum:
Ellipsoid: Du Plessis 1817, where a=6376523 and 1/f=308.6409971
Origin: 49° 30`N and 7° 44`13.95"E

The GPS system has a global datum:
Ellipsoid: WGS84, with a=6378137 and 1/f=298.257223563
Origin: Earth center

The WGS84 ellipsoid is designed to best-fit the earth as a whole, the local mapping ellipsoid gives a better fit for the area it was designed for. The fact that different map datums have been used over time means that the latitude and longitude coordinate of a position on the earth's surface is NOT unique, it depends on the datum! With Google Earth, which is based on the WGS84 datum, you can see that the prime meridian of zero longitude is shifted over more then 100 m of the Observatory in Greenwich. This means that to draw the GPS position of the user on the map the local Map datum is not enough, besides we need the datum shift (displacement and orientation) relative to the global WGS84 datum. There is even more, because of tectonic movements this datum shift must be determined periodically.

Datum Shift
OziExplorer supports many datum shifts but not the one for our map serie. However it is possible to define a user datum shift. This can be done by defining the parameters in the datums.dat file in the OziExplorer directory. The European Petroleum Survey Group (EPSG) publish many datum shift parameters. There are multiple transformation methods with different parameters to shift the map datum to the WGS84 datum.

Molodensky-Badekas with 10 parameters:
- 3 to define the displacement (tx, ty, tz),
- 3 to define the rotation (rx, ry, rz),
- 3 to define the origin of rotation (xp, yp, zp),
- 1 to define the the scale factor k.
  Example: RD datum shift (Netherlands)
- tx, ty, tz : 593.0248, 25.9984, 478.7459 (meters)
- rx, ry, rz : 1.9342, -1.6677, 9.1019 (10^-6 rad)
- xp, yp, zp : 3903453.1482, 368135.3134, 5012970.3051 (meters, Amersfoort)
- k: 4.0725 (ppm)

Bursa-Wolfe with 7 parameters:
- the origin of rotation is the center of the earth (0,0,0).
- dx, dy and dz have other values than the previous method,
- dx, dy and dz are dependend of the rotation parameters, this is why the previous method is better.
  Example: RD datum shift (Netherlands)
- tx, ty, tz : 565.4171, 50.3319, 465.5524 (meters)
- rx, ry, rz : 1.9342, -1.6677, 9.1019 (10^-6 rad)
- k: 4.0725 (ppm)

Molodensky with 3 parameters
In this method only the 3 displacement parameters of the Molodensky-Badekas method are used (593, 26, 479). This method is used by OziExplorer.

Rotations are defined in radians, or in seconds, multiplied with a factor, mostly 1000000. There are two standards for rotations, the "Position Vector" and the "Coordinate Frame", the difference is the sign of the values. Sometimes the datum shift is defined inclusive the relative values compared to the WGS84 reference ellipsoid da and df, the value for df is often multiplied by a factor, usually 10000.

The datum shift for our maps could not be found on the Internet. This means we have to calculate them and add them in the datums.dat file with Name, id, tx, ty, tz. We name our Map datum NDG (Nord de Guerre), the id of the Du Plessis ellipsoid in OziExplorer is 28.

Calculate datum shift parameters
The datum shift values can be calculated with the lat/lon coordinates of minimal 3 reference points for both datums (NDG map datum and WGS84 GPS datum). Reference points which can be clearly identified in the map and in Google Earth are for instance bridges and rail crossings. OziExplorer can be used to get waypoints from both resources. I choose several points on multiple maps for a acceptable precise determination. During the calculation I skip the points which show a bad fit, these points could have been changed during 65 year.

First we digitize the waypoints in the local datum of the maps and save them in a waypoint file, then we digitize the same waypoints in Google Earth. This can be done with OziExplorer by loading the Google Earth map, the necessary map file can be loaded from the OziExplorer website. After the waypoint file is saved it can be used for calculating the datum shift.
To calculate the datum shift the shareware program Sevenpar of Killet Soft can be used. With Sevenpar the 7 parameters of the Bursa-Wolfe and the 3 parameters of Molodensky can be calculated with many reference points to get precise results. Another option is to use the open source ILWIS add-on Inverse Molodensky. By using this program you must define the lat/lon coordinates of the reference points in seconds. Inverse Molodensky is limited to maximal 3 reference points, by calculating multiple sets of 3 points you can achieve reasonable result by averaging.

The results of my calculations gives the following line in datums.dat: NDG, 28, 1383, 44, 454. The standard deviation of the results were 7 m for tx, ty and tz.

This completes the data we need to calibrate the mapserie M831.

Finishing touch
The calibration setup must be updated with the new Map datum NDG. Don't forget to copy the updated datums.dat file to your OziExplorerCE device.
With the Img2Ozf utility of OziExplorer you can generate tiles from the image with multiple zoomlevels, this will give you a better performance and better quality. For OziExplorerCE it is a must.
With these digital maps you can walk or drive back in time 65 years ago, that is fun. With the freeware program OziMaptoKMZ you can generate an overlay for Google Earth, with the slider you can define the transparancy of the map, it's nice to see what is changed in 65 years.

What was not possible
Unfortunately it is not possible with OziExplorer to display the X,Y grid or to calculated the X,Y coordinates for this projection. On the border of the map the False Coordinates of Origin are given: 600,000m N and 300,000m E. In the Projection Setup menu it is not possible to define these values. In OziExplorer only a few grids are supported: UTM and user defined grids (Transverse Mercator). For the Dutch maps it is possible in this way to define a user RD grid. On the OziExplorer website we can read at Future Plans that support for more local grids will be extended.
By the way I found that two ways are used to define False Northing, related to the equator or related to the origin.


Lambert Conformal Conic 1SP TO 2SP Conversion

Ellipsoid		1SP		To	2SP
1/f		lat origin
		scale factor

Note: This is a basic script to convert Lambert 1SP to 2SP parameters.
The prepopulated values are for the US AMS mapserie M831.