Thanks for this detail Doug, I will try again. Your analysis is spot on as I did disable SSL temporarily to see if the websockets would in fact work that way confiming it is an SSL issue. If you go to unsecured http (http://shakerweather.com) you should see it all working with websocket updates (until I go back in and try to fix it the correct way with SSL this afternoon)
Light at the end of the tunnel. I will post later today when I either reach success or get stuck again. On Saturday, July 24, 2021 at 1:05:12 PM UTC-4 [email protected] wrote: > I just checked your site in edge and pulled up the dev console. I see the > main issue is that the client (me) is trying to access a non-secure > websocket (your server) while the rest of the content is coming from the > https stream: > > *paho-mqtt.min.js:37 Mixed Content: The page at > 'https://shakerweather.com/ <https://shakerweather.com/>' was loaded over > HTTPS, but attempted to connect to the insecure WebSocket endpoint > 'ws://mqtt.shakerweather.com:9001/mqtt > <http://mqtt.shakerweather.com:9001/mqtt>'. This request has been blocked; > this endpoint must be available over WSS.* > > I had this problem before and what I to solve it was to have CloudFlare to > translate all my traffic to SSL. > > Here is my Cloudflare and NGINX Proxy Manager setup that I use for > largoweather.com. Essentially if you check my SSL Cert, it is provided by > Cloudflare. Cloudflare pushes all http traffic as https to my internal > server. I have firewall rules to route all incoming 80/443 traffic to my > internal server in which that is properly routed to my NGINX Proxy Manager > instance. > > NGINX Proxy manager proxies that traffic to the correct webserver (in my > case container) that hosts my weewx website. I did issue a LetsEncrypt SSL > within NGINX Proxy manager to get that to work. > > They trick with LetsEncrypt is that you need to expose (just for a second) > your public IP Address in Cloudflare (assuming that is your name server of > the domain) for the SSL Certificate to be created correctly. Once you do > that, you can setup Proxy again to protect your public IP. > > Once this is setup, you will immediately see the live updates working. > > On Saturday, July 24, 2021 at 12:36:18 PM UTC-4 Doug Jenkins wrote: > >> Matt: >> >> I am looking into the issue now. I think you need to offload the SSL to >> Cloudflare with NGNIX Proxy. I am capturing screenshots of my setup to see >> if that will help you. >> >> Doug Jenkins >> >> On Saturday, July 24, 2021 at 12:33:22 PM UTC-4 [email protected] >> wrote: >> >>> Doing more trouble shooting. I have narrowed it down to a SSL issue. >>> >>> To confirm this I disabled SSL at cloudflare and disabled force SSL for >>> shakerweather.com with my NGINX reverse proxy server and ran with this >>> setup: >>> >>> *weewx.config* >>> [[MQTT]] >>> server_url = mqtt://usr:[email protected]:1883/ >>> >>> topic = weather >>> unit_system = US >>> binding = archive, loop >>> aggregation = aggregate >>> >>> *skin.conf* >>> # MQTT Websockets defaults >>> mqtt_websockets_enabled = 1 >>> mqtt_websockets_host = "mqtt.shakerweather.com" >>> mqtt_websockets_port = 9001 >>> mqtt_websockets_ssl = 0 >>> mqtt_websockets_topic = "weather/loop" >>> disconnect_live_website_visitor = 1800000 >>> >>> *mosquitto.conf* >>> persistence false >>> >>> # mqtt >>> listener 1883 >>> protocol mqtt >>> >>> # websockets >>> listener 9001 >>> protocol websockets >>> >>> allow_anonymous true >>> password_file /etc/mosquitto/passwd >>> acl_file /etc/mosquitto/acl >>> >>> >>> I am able to see the real time websocket updates internally and >>> externally at http://shakerweather.com (I'll leave it up for a short >>> time so if it doesn't work if you click on it that could be why) >>> >>> So at least I have narrowed down the issue. Still not sure how to >>> resolve to ensure all is secured with SSL.... >>> On Saturday, July 24, 2021 at 10:23:03 AM UTC-4 Matt Johnson wrote: >>> >>>> Your setup seems similar to mine so I went for it. Still not working. >>>> >>>> I went ahead and added a mqtt CNAME record to the shakerweather.com >>>> cloudflare setup. I also added mqtt.shakerweather.com to my NGINX >>>> Proxy Server to forward to port 9001 using the same SSL certificate I have >>>> for shakerweather.com that is forwarding to port 80 to serve the page. >>>> >>>> Cloudflare: >>>> TYPE NAME CONTENT >>>> CNAME mqtt shakerweather.com >>>> A shakerweather.com static wan IP >>>> >>>> *weewx.conf* >>>> [[MQTT]] >>>> server_url = mqtt://user:[email protected]:1883/ >>>> <http://user:[email protected]:1883/> >>>> >>>> topic = weather >>>> unit_system = US >>>> binding = archive, loop >>>> aggregation = aggregate >>>> >>>> *skin.conf* >>>> # MQTT Websockets defaults >>>> mqtt_websockets_enabled = 1 >>>> mqtt_websockets_host = "mqtt.shakerweather.com" >>>> mqtt_websockets_port = 443 >>>> >>>> mqtt_websockets_ssl = 1 >>>> mqtt_websockets_topic = "weather/loop" >>>> disconnect_live_website_visitor = 1800000 >>>> >>>> *mosquitto.conf* >>>> persistence false >>>> >>>> # mqtt >>>> listener 1883 >>>> protocol mqtt >>>> >>>> # websockets >>>> listener 9001 >>>> protocol websockets >>>> >>>> >>>> allow_anonymous true >>>> password_file /etc/mosquitto/passwd >>>> >>>> acl_file /etc/mosquitto/acl >>>> On Saturday, July 24, 2021 at 8:33:43 AM UTC-4 [email protected] >>>> wrote: >>>> >>>>> I had a number of struggles with this when I setup the Belchertown >>>>> skin on my Raspberry Pi hosting my weather site, www.largoweather.com. >>>>> I think the issue is that you need your mqtt_websockets_port to be set to >>>>> 443 as the websocket traffic is getting filtered out by your firewall >>>>> (9001->8083) >>>>> >>>>> Here is what I did to get this to work on largoweather.com: >>>>> >>>>> 1. Setup Cloudflare to manage the DNS proxy on 2 domains: >>>>> - largoweather.com (A Record) >>>>> - wx.largoweather.com (aka your mqtt.beldenserver.com) - CNAME >>>>> Record pointing the content to largoweather.com >>>>> - Setup my SSL/TLS to Strict. I am using Cloudflare to offload my >>>>> SSL processing so essentially all traffic is coming in through 443. >>>>> >>>>> These are pointed to my public IP address which is dynamic. I use a >>>>> shell script to update cloudflare' content to keep my public IP current >>>>> for >>>>> their system >>>>> >>>>> 2. NGNIX Proxy Manager : I use this program to manage my NGINX >>>>> instance that acts as a reverse proxy manager for my domains: >>>>> 2.1 : Setup: >>>>> - largoweather.com : Setup to point to my local server ip >>>>> address. The scheme is http. I use the program's lets encrypt function to >>>>> get a SSL certificate and force SSL traffic to my final weewx website. >>>>> - wx.largoweather.com : This handles my sockets setup. I >>>>> forward all traffic to port 9001 and use the same ssl certificate issued >>>>> for largoweather. I force SSL on this setup as well. >>>>> >>>>> 3. mosquitto configuration : I setup mosquitto on the same server as >>>>> my weewx install since everything is running on the pi. here is my >>>>> mosquitto.conf: >>>>> pid_file /var/run/mosquitto.pid >>>>> persistence true >>>>> persistence_location /var/lib/mosquitto/ >>>>> log_type error >>>>> websockets_log_level 1023 >>>>> connection_messages true >>>>> log_dest file /mnt/*****/weewx/logs/mosquitto.log >>>>> >>>>> allow_anonymous true >>>>> password_file /etc/mosquitto/passwd >>>>> acl_file /etc/mosquitto/acl >>>>> >>>>> listener 9001 >>>>> protocol websockets >>>>> >>>>> listener 1883 >>>>> protocol mqtt >>>>> log_type error >>>>> >>>>> >>>>> 4. weewx/Belchertown configuration: Here I setup my MQTT to talk to my >>>>> local IP address on port 1883. Remember the traffic is all coming in on >>>>> port 443, so that is the port I need Belchertown skin to essentially >>>>> connect to resolve the web sockets requests. >>>>> >>>>> [[MQTT]] >>>>> server_url = >>>>> mqtt://joeuser:xxxxxxx@<YOUR-LOCAL-SERVER-IP>:1883/ >>>>> topic = weather >>>>> unit_system = US >>>>> binding = archive, loop >>>>> aggregation = aggregate >>>>> [[Belchertown]] >>>>> skin = Belchertown >>>>> enable = True >>>>> >>>>> [[[Extras]]] >>>>> site_title = Largo Weather >>>>> mqtt_websockets_enabled = 1 >>>>> mqtt_websockets_host = wxsocket.largoweather.com >>>>> mqtt_websockets_port = 443 >>>>> mqtt_websockets_topic = weather/loop >>>>> mqtt_websockets_ssl = 1 >>>>> disconnect_live_website_visitor = 1800000 >>>>> >>>>> I hope this helps! >>>>> >>>>> Doug Jenkins >>>>> >>>>> >>>>> On Saturday, July 24, 2021 at 8:07:23 AM UTC-4 [email protected] >>>>> wrote: >>>>> >>>>>> Thanks Les. I think you have helped clear some things up. For the >>>>>> sake of clarity and getting to the end state I desire and your >>>>>> suggestion >>>>>> let's focus on locally hosting (my first setup) over the cloud broker. I >>>>>> have a robust homelab and really like to keep things in house. >>>>>> >>>>>> *think the first configuration, with the local mqtt broker isn’t >>>>>> going to work because mqtt_websockets_host is set to localhost, which >>>>>> will >>>>>> only resolve to the weewx/mqtt server when the web browser is running on >>>>>> that server. You need to set something here that will resolve to the >>>>>> weewx/mqtt server from any client that you want to get realtime updates. >>>>>> * >>>>>> >>>>>> This makes sense. I initially was on this path and changed the >>>>>> mqtt_websockets host to the ip address of the weewx/mqtt server. This >>>>>> did >>>>>> not work for other clients on the LAN which seemed strange given how I >>>>>> have >>>>>> lots of locally running services on a few different servers in my home >>>>>> lab >>>>>> and I access them all via device IP and port, with a few that are >>>>>> accessible externally via reverse proxy. >>>>>> >>>>>> >>>>>> *You need a DNS name that will resolve to your firewall and get >>>>>> port-forwarded (for port 8083) to the weewx/mqtt server, >>>>>> for mqtt_websockets_host. That should enable external access. * >>>>>> >>>>>> I can use my mqtt.beldenserver.com DNS name I have setup at >>>>>> cloudflare for this. I am running OPNSense for my firewall so I should >>>>>> be >>>>>> able to do anything. Right now I have a firewall rule setup to send port >>>>>> 80/443 traffic to my NGINX Reverse Proxy where I have several DNS >>>>>> addresses >>>>>> pointing to different services on different servers. One of these takes >>>>>> shakerweather.com through 80/443 and through the reverse proxy and >>>>>> points to the weewx/mqtt server to return the webpage (which is working >>>>>> fine) >>>>>> >>>>>> So you are saying I just need to add a rule to forward external port >>>>>> 8083 requests to the weewx/mqtt server IP and port? >>>>>> >>>>>> *And, if your firewall will do hairpinning, it should work internally >>>>>> as well. It may take some magic with forwarding/masquerading rules on >>>>>> the >>>>>> firewall to get hairpinning to work. (The alternative for internal >>>>>> access >>>>>> is to have an internal DNS server that resolves that hostname directly >>>>>> to >>>>>> the internal IP of the weewx/mqtt server for clients on the internal >>>>>> network.)* >>>>>> >>>>>> My OPNSense firewall should be able to do hairpinning. I read on that >>>>>> briefly as I have only heard of the term and not too familiar with it. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Saturday, July 24, 2021 at 2:24:32 AM UTC-4 ln77 wrote: >>>>>> >>>>>>> I think the first configuration, with the local mqtt broker isn’t >>>>>>> going to work because mqtt_websockets_host is set to localhost, which >>>>>>> will >>>>>>> only resolve to the weewx/mqtt server when the web browser is running >>>>>>> on >>>>>>> that server. You need to set something here that will resolve to the >>>>>>> weewx/mqtt server from any client that you want to get realtime >>>>>>> updates. >>>>>>> >>>>>>> Not sure why the second config, with the cloud mqtt broker, isn’t >>>>>>> working. Are you sure the mqtt broker is configured for SSL on port >>>>>>> 8883? >>>>>>> You might put “log_success = true” in the weewx [[MQTT]] config and >>>>>>> see if >>>>>>> the log messages tell you anything useful. >>>>>>> >>>>>>> Or forget about the cloud server and go back to getting the first >>>>>>> config working. You need a DNS name that will resolve to your firewall >>>>>>> and >>>>>>> get port-forwarded (for port 8083) to the weewx/mqtt server, >>>>>>> for mqtt_websockets_host. That should enable external access. And, if >>>>>>> your >>>>>>> firewall will do hairpinning, it should work internally as well. It may >>>>>>> take some magic with forwarding/masquerading rules on the firewall to >>>>>>> get >>>>>>> hairpinning to work. (The alternative for internal access is to have >>>>>>> an >>>>>>> internal DNS server that resolves that hostname directly to the >>>>>>> internal IP >>>>>>> of the weewx/mqtt server for clients on the internal network.) >>>>>>> >>>>>>> -Les >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 23 Jul 2021, at 21:05, Matt Johnson <[email protected]> wrote: >>>>>>> >>>>>>> I've been trouble shooting getting the Belchertown skin MQTT >>>>>>> Websocket real time updates to work on my site shakerweather.com >>>>>>> for a lot of this week. >>>>>>> >>>>>>> I have WeeWx installed on a dedicated thin client on Ubuntu 20.04 >>>>>>> LTS and have Mosquitto and NGINX installed on the same machine. Running >>>>>>> bare metal, no docker or VMs here. >>>>>>> >>>>>>> External access to WeeWx website is handled via NGINX reverse proxy >>>>>>> manager with SSL certs on a different server via docker. Requests to >>>>>>> shakerweather.com are sent to the proxy server and then to the >>>>>>> WeeWx machine. >>>>>>> >>>>>>> With this setup, the site is served up fine internally and >>>>>>> externally with the updates at archive intervals every 5 minutes. >>>>>>> >>>>>>> I know that the weewx-mqtt extension is installed correctly as I >>>>>>> have been able to test it locally and get the websocket updates to work >>>>>>> perfectly with the following configs: >>>>>>> >>>>>>> *weewx.conf* >>>>>>> [[MQTT]] >>>>>>> server_url = mqtt://user:pw@localhost:8883/ >>>>>>> topic = weather >>>>>>> unit_system = US >>>>>>> binding = archive, loop >>>>>>> aggregation = aggregate >>>>>>> >>>>>>> *skin.conf* >>>>>>> # MQTT Websockets defaults >>>>>>> mqtt_websockets_enabled = 1 >>>>>>> mqtt_websockets_host = "localhost" >>>>>>> mqtt_websockets_port = 8083 >>>>>>> mqtt_websockets_ssl = 0 >>>>>>> mqtt_websockets_topic = "weather/loop" >>>>>>> disconnect_live_website_visitor = 1800000 >>>>>>> >>>>>>> I am only able to see the the real time updates on the local machine >>>>>>> only with WeeWx and Mosquitto. If I try to access it by IP address >>>>>>> elsewhere on my LAN on other clients it does not connect and eventually >>>>>>> fails. No luck externally either - despite my NGINX Reverse Proxy >>>>>>> Manger >>>>>>> handling serving the page and SSL certs the websocket real time updates >>>>>>> don't pass through. That was my original thought of how it would work. >>>>>>> >>>>>>> After much trial and error, and reading every thread imaginable on >>>>>>> this along with many messages and some correspondence with Pat O'Brien >>>>>>> I >>>>>>> decided to go ahead and setup a Digital Ocean Ubuntu VM and install >>>>>>> Mosquito there to serve as a cloud broker. I followed Pat's >>>>>>> instructions >>>>>>> exactly as he outlines in setting up the cloud broker: >>>>>>> https://obrienlabs.net/how-to-setup-your-own-mqtt-broker/ >>>>>>> >>>>>>> I have the cloud MQTT broker installed correctly at Digital Ocean >>>>>>> with Let's Encrypt, and ran tests on it. Messages can be sent when >>>>>>> authenticated, ports are open, etc. However, I can get no further with >>>>>>> the >>>>>>> websockets real time updates than "Connected. Waiting for data". If I >>>>>>> reboot the cloud MQTT broker I immediately get a disconnected message >>>>>>> on >>>>>>> the website so it does appear to be connecting and waiting for data. >>>>>>> Somehow the data is simply not transferring from my WeeWx client to the >>>>>>> cloud MQTT broker at Digital Ocean. The other weird thing is if I try >>>>>>> to >>>>>>> access shakerweather.com or the website by local IP address on the >>>>>>> machine that hosts WeeWx I always get a failed message, won't even >>>>>>> connect >>>>>>> to the server. However, any other client on my LAN and external on WAN >>>>>>> does >>>>>>> not have this issue. >>>>>>> >>>>>>> Here are my current configs: >>>>>>> >>>>>>> *weewx.conf* >>>>>>> [[MQTT]] >>>>>>> server_url = mqtt://user:[email protected]:8883/ >>>>>>> topic = weather >>>>>>> unit_system = US >>>>>>> binding = archive, loop >>>>>>> aggregation = aggregate >>>>>>> [[[tls]]] >>>>>>> tls_version = tlsv1 >>>>>>> ca_certs = /etc/ssl/certs/ca-certificates.crt >>>>>>> >>>>>>> *skin.conf* >>>>>>> # MQTT Websockets defaults >>>>>>> mqtt_websockets_enabled = 1 >>>>>>> mqtt_websockets_host = "mqtt.beldenserver.com" >>>>>>> mqtt_websockets_port = 8083 >>>>>>> mqtt_websockets_ssl = 1 >>>>>>> mqtt_websockets_topic = "weather/loop" >>>>>>> disconnect_live_website_visitor = 1800000 >>>>>>> >>>>>>> At this point, I have spent 20+ hours on this and hoping someone >>>>>>> here can point me in the right direction, it seems data is just not >>>>>>> feeding >>>>>>> the MQTT topic. I'm fine with using Digital Ocean as a cloud MQTT >>>>>>> server >>>>>>> just to get it up and running. My preferred state is eventually to >>>>>>> selfhost >>>>>>> it all. >>>>>>> >>>>>>> Thanks in advance for anything I may be overlooking, advice or >>>>>>> possible solutions. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "weewx-user" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/weewx-user/38183ff6-d5d5-4151-a1b3-93fd618aef5cn%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/weewx-user/38183ff6-d5d5-4151-a1b3-93fd618aef5cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e3f9b02b-8512-446d-b5b8-9a3991269f08n%40googlegroups.com.
