Network Traffic Analysis With DD-WRT Netflow and Ntop

Introduction

DD-WRT include the capability of running rflow, a Cisco Netflow data exporter implementation. The netflow data is sent to a port of a computer (management server) on your LAN running a Netflow collector, in this case this is ntop.
Ntop is an open source network traffic monitoring tool that shows the network usage via a web browser. The web interface for monitoring, configuration and administration make ntop easy to use and suitable for monitoring various kind of networks. 

Router Setup

My router is running the DD-WRT v24-sp2 firmware (you need a version with rflow support).
Rflow can monitor the available interfaces of the router, in my case these are br0 (Lan & Wlan), vlan0, eth1, Wlan0 and WAN.
Log into your router through your browser, than go to the Services | Services where you will find the RFlow / MACupd section.
RFlow: select "Enable"
Server IP: The IP address of your computer that wil run ntop.
This computer must have a static IP address, or using a DHCP static lease.
Port: The UDP port that will be used to send the netflow information.
Common default ports for Netflow are 2055 and 9996. In my first setup I used port 2055 but because this port was used by other applications I had to change to 9996 to make my installation stable.
Interface: LAN&WLAN
Then click "Apply Settings"

Setup ntop management server

Ntop will run on many operating systems. The ntop Web site offers multiple versions (sources and binaries) of the package for download. The free binary version for Windows is limited to capturing only the first 2000 packets. This limitation does not exist in the Linux versions and the paid Windows version. I use the free unlimited version ntop for win32 v3.2 from OPENXTRA. You can find the easy to install OPENXTRA version on several places on the Internet like here [1]
To run ntop for win32 I use an old Windows XP laptop (320MB RAM) as terminal-, web-, file-, print- and scanner server in my network and this laptop proofs to have enough resources to do this additional job.
After the installation is finished, you should have a new icon in the system tray called OPENXTRA Commander. Double-click this icon to open the OPENXTRA Commander. If the NTop Service plug-in is not started, click Start in the Action column to start it. Once it is started, click the Launch action for the NTop plug-in, which will open your browser (http://localhost:3000). If all is well, you will already be collecting some impressive data. In the Windows Control panel | Administrative Tools | Services you can check if the ntop service is running. Optional you can configure the service to restart automatic after a failure.

Rfow configuration

You have to create a virtual rflow interface. Do this by selecting Plugins | All in the menu listing at the top of the webpage.
In the Active column click on "NO" next to NetFlow to enable the plugin.
Click on "NetFlow" in the Configure column.
Click on "Add NetFlow Device".
NetFlow Device Name: Any name you like, I choose DD-WRT.
Click on "Set Interface Name".
Local UDP Collector Port: Use the same port as configured in the router (I used 9996).
Click on "Set Port".
Virtual Netflow Interface Network Address: Your LAN network address and its netmask.
If your router is using 192.168.1.1, then this should be: 192.168.1.0/255.255.255.0. Ntop uses this address to recognize the local hosts from the remote hosts.
Click on "Set Interface".
Leave everything else now to defaults.
Now we have two interfaces ntop can monitor, the NIC of the local computer and the Netflow interface. We now can switch which interface we wish to monitor.
In the menu at the top select Admin | Switch NIC.
Under Available Network Interface select the NetFlow Device name you entered earlier (DD-WRT in my case).
Then click on Switch to NIC.
You will notice that some menu entries have a small padlock icon in them; these are the Web pages that require a password to access. The default credentials for the XTRA package is user = admin, password = admin.
You can take a first view of the web pages of ntop.

Rrd configuration

ntop stores all of its active data in RAM, so if the system is reset, you lose all your data. Archiving of this data to disk must be configured. Ntop is using rrd to prevent that your disk fills up. RRD uses a round robin database which stores time-series data in a very compact way so that it will not expand over time. You have to configure what data with what detail must be archived and with what retention time.
Do this by selecting Plugins | All in the menu listing at the top of the webpage.
In the Active column click on "NO" next to rrdPlugin to enable the plugin.
Then click on "rrdPlugin" in the Configure column to show the RRD Preferences.
I let the first 6 items (interval to delay) unchanged.
Data to dump: Hosts and Interfaces
Hosts Filter: I choose to only archive data of local hosts with my LAN address 192.168.1.0/255.255.255.0
RRD detail: I choose medium
RRD Files Path: This is the location where rrd stores his data: \NTopWin32\rrd.

Ports

Ntop will tell you on different pages which ports are used. In the Windows version you will find the file "services" (without extension) in the directory C:\Program Files\OPENXTRA\NTopWin32.
The structure of the file is

port         port         alias        # description
name         number/tcp  

like this:

pop2         109/tcp   pop-2        # Post Office Protocol - V2
pop3         110/tcp   pop-3        # Post Office Protocol - Version 3
sunrpc       111/udp   rpcbind

You can add portnames to this file if you want ntop to show a portname 

in stead of a port number. I have added the follwing ports:

#
# Extra services
#
ssh          22/tcp
ms-sql-s     1433/tcp
ms-sql-s     1433/udp
ms-sql-m     1434/tcp
ms-sql-m     1434/udp
upnp         1780/tcp
upnp         1780/udp
ssdp         1900/udp
ntop         3000/tcp
rdp          3389/tcp
rdp          3389/udp
atq          3456/udp
nat-t        4500/udp
remotescan   6077/tcp
remotescan   6078/udp
netflow      9996/udp
dropbox      17500/tcp
dropbox      17500/udp

Stop and start the ntop service for the changes to take effect.

Protocols

There is a default list of protocols ntop will monitor for you, if you want you can define a smaller list of protocols or you can add protocols to the list. To do this you have to create a protocol.list in the directory C:\Program Files\OPENXTRA\NTopWin32. The ntop web page will display the protocols in the same order as they are defined in the list, you can change the order if you like.
The structure of the file is protocolname=port, where port is a portname that you can find in the file services or a portnumber.
Behind the = you can define multiple ports with port|port| or with 12-20.
In the following file I have first defined the default values and added some extra protocols.

## Default ntop protocollist ##
FTP=ftp|ftp-data|69
HTTP=http|www|https|3128
DNS=name|domain
Telnet=telnet|login
NBios-IP=netbios-ns|netbios-dgm|netbios-ssn
Mail=pop-2|pop-3|pop3|kpop|smtp|imap|imap2
DHCP-BOOTP=67-68
SNMP=snmp|snmp-trap
NNTP=nntp
NFS/AFS=mount|pcnfs|bwnfs|nfsd|nfs|nfsd-status|7000-7009
VoIP=5060|2000|54045
X11=6000-6010
SSH=22

## Default ntop Peer-to-Peer protocols ##
Gnutella=6346|6347|6348
Kazaa=1214
WinMX=6699|7730
DC++=-1
eDonkey=4661-4665
BitTorrent=6881-6999|6969

## Default ntop Messenger protocols ##
Messenger=1503|1863|5000|5001|5190-5193

## Extra ntop protocols ##
Comodo=1037-1045|1280|4447-4448
Avast=1281-1282
NetFlow=9996
UPnP=1780|1900
Dropbox=17500
Ntop=3000
Remotescan=6077|6078
RDP=3389
Streaming=554|1755|1935|3689|4070|5222|7070
Nat-t=4500
IIS=1025|3456
SQL=1433
LDAP=ldap|ldaps
RPC=111
SLP=427
LPR=515|631

After you have created the file you have to configure ntop to use this file. Select in the menu listing at the top of the webpage Admin | Configure | Startup Options and select the IP Preferences link. In the field TCP/UDP Protocols To Monitor (-p) specify C:\Program Files\OPENXTRA\NTopWin32\protocols.list (must be the full path). See ntop internal help for more information. Stop and start the ntop service for the changes to take effect.

Usage

In the Ntop Bandwidth Monitoring Guide you can find some interesting usage scenario's, like:

• Who are the top internet bandwidth users on my network?
• What websites do the top bandwidth users visit?
• What websites get the most traffic from within my organization?
• What websites' traffic consumes most of my bandwidth?
• What applications are being used?
• Which local hosts share the most data?
• At what time of the day is the network most Utilized?
• Performing a network inventory
• Exporting traffic data
• Detecting network security violations?

Suppose you identify a particular host as the major consumer of bandwidth, what if you want to find out just what exactly he is doing online that is consuming so much bandwidth? Here is how ntop can help:

1. Identify the host you are interested in [one way is to sort on the Data for Network Traffic stats for local hosts.
2. Click on that host to bring up the Info about xxxxx page where xxxx is the name or IP address of the host you are interested in.
3. Scroll down to the bottom of the page to the Active TCP/UDP Sessions table. A screen is shown which "lays it all out for you".

Host Fingerprints
You can switch the interface you want to monitor. You should remember when you use the NetFlow interface that NetFlow does not send you the actual packages like the local NIC interface. This is why ntop cannot report fingerprints when the Netflow interface is used. You can find Host Fingerprints in the menu IP | Local | Host Fingerprints.

Local Matrix
In the menu IP | Local | Local Matrix you will, when using the NetFlow interface, see no traffic between local hosts. This is caused by the default behaviour of a switch that is used to connect your local hosts to the router. Only the traffice between local hosts and remote hosts is captured.

Historical data
Historical data can be viewed with ntop (or other tools) by clicking on the icon on webpages like Info about Host and Plugin | Round Robin Databases | Arbitrary Graphs.

Dumping Ntop Data
There are scripts to dump data in a MySQL database on sourceforge.net. However, within ntop, just click Utils|Data Dump to show a dialog box. You can dump data about different objects into different formats – see the ntop guide for the formats. Some of these formats are importable into a spreedsheet and from there you can unleash the full power of Open Office Calc or Excel unto your traffic data.\

Workarounds

In this version of ntop there are a few small annoyances to work around.

Hostname resolving
PC's in my network are not always on, I found when ntop starts he resolves the hostnames of al host in the network, but after some time when these PC's switch off and on ntop forget the hostname. For me it was no problem to restart the ntop service with a bat file every night with a scheduled task of Windows. The bat file look like this

@echo off
REM - File: Daily Restart ntop for win32.bat
REM - Description: Restart ntop for Win32 Service tbv name resolving
REM - Author: Jan S
echo Restarting ntop for Win32...
echo ======================================================
net stop "ntop for Win32"
net start "ntop for Win32"
echo ======================================================
echo ntop for Win32 Restarted

Broken links
On several pages links are used to services on the Internet for additional information. There are links to WHOIS information and there is a link voor ASN information.
There are several ways to solve this problem, because I use Firefox the easiest way for me was url rewriting in the browser, you can use the Firefox Add-On Redirector to do that.

WHOIS was http://www.radb.net/cgi-bin/radb/whois.cgi?obj=*
can be rewritten to http://whois.domaintools.com/$1 or http://www.lookip.net/$1 or any other you like.

ASN was http://ws.arin.net/cgi-bin/whois.pl?queryinput=*
ca be rewritten to https://apps.db.ripe.net/search/query.htm?searchtext=$1
FAQ, in the menu About there is a link to FAQ but the faq.html file is not local available. You can copy the file from www.ntopsupport.com/faq.html to C:\OPENXTRA\NTopWin32\html\faq.html.
There is a lot of important information available here.

Finally

The combination of a DD-WRT router running rflow and a PC running ntop provides a low cost solution for remote network traffic usage and activity (NetFlow monitoring). Rflow provides fast packet capture and also captures packets efficiently thus preserving CPU cycles. With ntop Luca Deri has created a brilliant tool for seeing what is happening on your network in realtime. This is only a basic tutorial to show what you can do with DD-WRT and rflow. ntop has many more possibilities, out of the box via configuration and via extra scripts which are available in the directory C:\OPENXTRA\NTopWin32\www and on the Internet.
There is a new version 5 available for Windows with many more possibilities, it's worth looking at it.

External Links

• Configuring Ntop to work with the DD-WRT firmware for the Linksys WRT54G(S) router
• Configuring ntop
• Ntop Netflow with a WRT54GS Firewall/Router and NST Probe
• RFlow Collector and MySQL
• Ntop, persistent data and rrd
• Ntop Bandwidth Monitoring Guide
• Ntop Users Guide
• Ntop Tutorial Slides

This information is published earlier by me at DD-WRT.com

Disable NetBios, enable DNS with DD-WRT

Introduction

Using a LAN router based on DD-WRT software offers extra features and functionality sets to improve your network, easy and for low costs. In this article I explain the setup of DNS/DHCP for standard name resolution while disabling NetBIOS.

NetBIOS

NetBIOS is a legacy API from the early days (1983) of PC networking providing services for applications on seperate PC's to communicate in a single local PC network. NetBIOS uses therefore a flat namespace of NetBIOS names and makes extensive use of broadcasting for name resolving of adresses.
SMB is an application-layer network protocol mainly used for file- and print sharing. SMB can run on top of the Session (and lower) network layers in several ways:

• Via the NetBIOS API, which in turn can run on several transports:
• On several legacy protocols such as NBF, NBX and Pathworks. 
• On UDP ports 137, 138 & TCP ports 137, 139 (NBT);
• Directly over TCP, port 445 (Direct hosting of SMB over TCP/IP).

The first implementations of NetBIOS run directly on the Link Layer by the NBF protocol, later NetBIOS came available on routable networks to support existing NetBIOS based applications.

The support of legacy services to support old applications has resulted in complexity and limitations because of the use multiple Naming Systems (NetBIOS name and Hostname, NetBIOS Node types, WINS and DNS, LMHOSTS and HOSTS, .. ) and multiple network management utilities. Communication on Internet requires a hierarchical namespace and worldwide standards for addressing and network management. Because new applications do not depend on NetBIOS anymore, disabling NetBIOS and using the same standards is the way to go for:

• simplification, improvement security and speed,
• removing multiple methods of nameresolution,
• standardizing on TCP/IP name resolution on DNS for file and printer sharing.  

Impact of disabling NetBIOS
Shutting off NetBIOS reduces the network's browsing functionality because the Computer Browser - the service manifested in Network Neighborhood, My Network  Places, and the Net View command - sits atop NetBIOS. Connecting to a share doesn't change: 

• net view command to view resources of a host with hostname
• net use command to access a share
• using UNC name in Start, Run or in address bar of Internet browser
• connect a drive from My Computer or Windows Explorer
• add a Shortcut to your My Network Places folder

In our case users has predefined access to resources, so limited browsing functionality is not an issue, as network manager there are plenty of free tools available for browsing the network.
Disabling NetBIOS means that the infamous ports (UDP 137 and 138, TCP 137 and 139) are not used anymore, direct hosting of SMB over TCP/IP uses UDP en TCP port 445, you might have to update your firewalls for these changes.

Disable NetBIOS
To disable NBT manually, change the computer's TCP/IP properties. In the Network Connections window, you'll see an object for each network card on your system. Right-click the network card for which you want to disable NBT, then choose Properties. On the Properties page, double-click the Internet Protocol (TCP/IP) object, then click Advanced on the Internet Protocol (TCP/IP) Properties page. Click the WINS tab, then click the Disable NetBIOS over TCP/IP radio button. Clear the Enable LMHOSTS lookup check box, then click OK until you've closed the pages.
To verify that you've killed NBT, you can type "ipconfig /all" on a command line. You'll see a line confirming that NetBT is disabled. You can centrally disable NBT with DHCP, I will that explain later.

Disable and remove unnecessary services with care
The Computer Browser is of no use anymore. Type services.msc at the command prompt and select the Computer Browser to stop and disable the service.

The service TCP/IP NetBIOS Helper Service should, despite the name not be disabled. If you stop this service you will receive a system error 1231 with the message "The network location cannot be reached". On various websites it is erroneously recommended to stop this service. In fact it is a legacy name from the time SMB was related to NetBIOS, according to Bill Grant a better name would be TCP/IP SMB Helper.

In My Network Places I disabled Entire Network with the registry key "NoEntireNetwork".

DNS 

The Internet maintains two principal namespaces, the domain namespace and the IP address namespace. DNS maintains the domain namespace and provides translation services between it and the address namespace.
Unlike the flat namespace of Netbios, DNS has a hierarchical namespace, organized in subordinate levels (subdomains) of the DNS root domain. A hostname in DNS is a fully qualified domain name (FQDN), this is a name that is completely specified in the hierarchy of the DNS, having no parts omitted. The dot in a DNS name is required to force a DNS lookup.
The IP address namespace is logically recognized as consisting of two parts: the network prefix and the host identifier. The subnet mask or the CIDR prefix determines how the IP address is divided into network and host parts.

The DNS hostname of a Windows computer is based on the Computername and the DNS suffix: <comptername>.<dns suffix>., like barebone.example.com with example.com as suffix.
The Computername is set during the installations of Windows as a Computer property, the DNS suffix will be centrally managened with dnsmasq to get DNS hostnames for all devices, including mobile devices such as phones and tablets. When the computername is used the system will automatically append the DNS suffix.

Best practice for internal domain name
A domain name must be carefully planned, the best practice for an internal domain name is to use a sub-domain of an external registered domain. Only if you are sure not using an external domain in the future you could use for example .lan (not .local) as TLD that is what I use in the example configuration.

Dnsmasq

Introduction
Dnsmasq is a lightweight DNS and DHCP server available at the DD-WRT LAN Router. Dnsmasq accepts DNS queries and either answers them or forwards them to a real, recursive, DNS server. It also answers DNS queries for DHCP configured hosts.
The dnsmasq DHCP server supports dynamic and static address assignments and multiple networks. It automatically sends a sensible default set of DHCP options, and can be configured to send any desired set of DHCP options, including vendor-encapsulated options.

Our local network with the subnets 192.168.2.0/24 and 192.168.3.0/24 is shown in the figure, the router configuration for the two subnets are explained in a previous blog.

Prerequisites
All our network devices must have their own computername, and be configured to get their network configuration via DHCP. If you have some machines with static IP addresses DNSmasq will incorporate them as well based on the hardware (MAC) address, so you don't need to change them.

Configuration

De configuration of dnsmasq in DD-WRT consists of two parts: a Basic Setup and an Additional Setup at the Tab Services.
Let's start with the Tab Basic Setup in the webinterface of the DD-WRT router.
On this page we enable DNS and DHCP for the subnet 192.168.3.0 with dynamic IP adresses.
Select Use DNSMasq for DHCP, Use DNSMasq for DNS and DHCP-Authorative.

Under the TAB Services we select Services for the remaining settings.
Under DHCP Server choose LAN&WLAN, this is your internal domain, the Domain Name at the Basic Setup page is the external domain.
Under LAN Domain we definine our internal domain name, in our case its is lan.
Under static leases you define the MAC Addresses, the computername and static IP Address.
Under DNSMasq we enable both DNSMasq and Local DNS.

The remaining settings must be defined in Additional DNSMasq Options. The syntaxt of the settings you can find in the DNSMasq manual. My network have these settings:

dhcp-range=wan,192.168.2.0,static                          Enables DHCP for subnet 192.168.2.0 with static IP Addresses (for dynamic addresses you can define the start and end address). 
wan is the label of this subnet to reference to in the following dhcp-options.
By default dnsmasq sends some standard options to DHCP clients:
- netmask and broadcast address are set to the same as the host running dnsmasq,
- the DNS server and the default Gateway are set to the address of the machine running dnsmasq,
- if the domain option is set, it is send as connection specific suffix and DNS suffix search list.
With dhcp-option these values can be overridden and additional options set. 
dhcp-option=wan,3,192.168.2.1                               Default Gateway (3) as the ISP router 
dhcp-option=wan,6,192.168.2.3                               DNS Server (6)

ptr-record=3.2.168.192.in-addr.arpa,DD-WRT.lan    Return a PTR DNS record for the Lan router.
local=/lan/                                                              Queries from .lan are not send to upstream server.
dhcp-option=43,01:04:00:00:00:02                           Disable NetBIOS over TCP/IP, this requires that the DHCP client is configured Default in Advanced TCP/IP settings.
This DHCP option must be defined last for Windows XP client, see Microsoft KB953761.

Final check
You can check your configuration in the file dnsmasq.conf by entering a cat /tmp/dnsmasq.conf in the Command Shell at the Tab Administration Command.
With WinSCP you can manage your configuration in a user friendly explorer like interface.
With ipconfig /all you can check if the configuration of your clients are correctly set, special attention for the DNS suffix and the message that NetBIOS over TCP/IP is disabled.

That's it: I think dnsmasq is easy to use but powerfull software.

External Links

• NetBIOS on Wikipedia
• Server Message Block (SMB)
• Direct hosting of SMB over TCP/IP
• Live Without NetBIOS 
• Network Location Cannot be Reached
• DNS on Wikipedia
• DHCP Architecture
• Dnsmasq Manual

Setup an extra Router in your LAN

Introduction 

In our local network I use a Linksys WRT54GL as LAN Router behind the ADSL Modem/Firewall/Gateway of the Internet Service Provider (ISP). On the LAN router I have installed DD-WRT router firmware, this is free Open Source software. The rationale to add the LAN Router is using the extra features and functionality set to

• enhance security, speed and understandability
• modernize and standardize with easy central management at low cost 

In this article I explain the configuration of the routers to create transparant communication between hosts at both sites of the router. In later blogs I will explain the configuration of other functions. Our local network is shown in the figure below:

The server provides terminalservices, file services, print services, scan services and management services. The server is based on Windows XP SP3 with only 320 MB memory, the clients are based on Windows XP, Vista and Windows 7, the mobiles are Android Tablets and Phones and iPhones. The Wireless AccessPoint is configured as a bridge at a central point in the building. I still use Windows XP with limited resources to force myself to efficiently configure the server.

IP Routing

Hosts and networks
IP adressing is based on the concept of hosts and networks. A host is essentially anything on the network that is capable of receiving and transmitting IP packets on the network, such as a PC, a Server or a Router.
The hosts are connected together by one or more networks. An IP address is 32 bits wide, and as said, it is composed of two parts: the network number, and the host number (like zip code and house number).
The subnet mask or the CIDR prefix determines how the IP address is divided into network and host parts. By convention, it is expressed as four decimal numbers separated by periods, such as 192.168.2.1.
In our case the network number consists of the first three numbers (192.168.x), this is defined with the netmask 255.255.255.0 or with the 192.168.x.0/24 CDIR notation where 24 is the number of bits of the network number. In our local network we use two (sub)networks with the IDs: 192.168.2.0 and 192.168.3.0. The host numbers per network can range from 1 to 254, 255 is a broadcast address.

Use of ARP
Each host has a hardware (or MAC) address, which is six bytes long, this is a unique identifier assigned to network interfaces. They are normally written in hexadecimal form separated by dashes (02-FE-87-4A-8C-A9 for example).
Suppose the laptop on the network 192.168.3.0 wants to send a packet to the server for the first time, and it knows the IP address of the server. To send a packet, the laptop needs to know the hardware address of the server. The Address Resolution Protocol (ARP) is used for dynamic discovery of this address. ARP caches IP addresses and corresponding hardware adresses he has discovered. ARP wil broadcast a request when the IP address is not yet in the cache.
In the case the laptop on the network 192.168.2.0 wants to send a packet to the server on the 192.168.3.0 network the Router must forward the packet from the 192.168.2.0 network to the 192.168.3.0 network. This is accomplished by having the laptop use the hardware address of the Router and the IP address of the server. The Router will forward the packet to the hardware address of the server. These hardware addresses are obtained using ARP as described earlier. Hosts that cannot be reached on the same network must be forwarded by a Router.

Routing
In the IP configuration of each host the IP address of a Default Gateway is defined, this Gateway is the router that forwards packets to other networks.
In the case of the 192.168.2.0 network there are two Routers, the LAN Router will forward the packets to the 192.168.3.0 network and the ISP Router will forward the packets for other networks on the Internet. This means that on the 192.168.2.0 network, besides the Default Gateway to the Internet, the Route to the 192.168.3.0 network must me defined.

Router Configuration

Configuration of the ISP Router
The route from the 192.168.2.0 network to the 192.168.3.0 network must be configured as a Static Route in the Router of the ISP with [Destination IP=192.168.3.0, Gateway=IP address LAN router].

When the laptop in the 192.168.2.0 network wants to send a packet to the server in the 192.168.3.0 network, the laptop sends it to the Default Gateway, this is the Router of the ISP, this Router forward the packet to the LAN Router, who will forward the packet to the server.
When the laptop on the 192.168.3.0 network sends a packet to a host on the Internet, the laptop sends the packet first to his Default Gateway, this is the LAN Router, who forwards the packet of his Default Gateway, this is the ISP Router, who forwards the packet to the host on the Internet. When a packet returns from the Internet, the ISP router knows the Route to the server through the Gateway defined in the Static Route.

Configuration of the LAN Router
The WAN port of the LAN router must be connect to a LAN port of the ISP router.
Most commodity routers has two Operating Modes, Gateway or Router. By default the router is configured with the operating mode Gateway. Gateway mode is intended for the connection to the Internet, Gateway mode gives you firewall and network address transalation (NAT) functionality which makes the LAN addresses invisible on the WAN side. The ISP Router must keep the default Gateway operating mode setting but for the LAN router the operating mode must be changed to Router to make network addresses visible from both sides of the router. You should also want to disable the firewall on the LAN router to allow full communication between the local networks.

As was described earlier the Default Gateway of the LAN router is the ISP Router.
With these changes we are able to communicate to devices throughout the network and have added new features and functionality.