Weather forecast data is available from a range fo online services with different patterns which can generally be accessed via an API call from Python. In this article we’ll explore how to extract the weather data from openweathermap.org and then display this on a map so that you can get a visual view of weather locations. This can be useful for a range of applications or as an additional widget for your next web project.
Hereby, we’re going to use python3 to get all the data about weather forecast. For this, various weather APIs can be bring into our purpose some of which are listed below:-
Using OpenWeatherMap to get weather forecasts in Python
First thing first, I’m using here one of the convenient and freely available openweathermap that provides weather data, including current weather data, forecasts, and historical data to the developers. It provides an API with JSON, XML and HTML endpoints and a limited free usage tier.
Users can request current weather information, extended forecasts and graphical maps (showing cloud cover, wind speed, pressure and precipitation).
Step 1: Get API Key from Open Weather Map
To use this current weather data API, one must need the API key, which can be obtained from https://openweathermap.org/api
See options for the API keys. For this tutorial I chose the Climatic Forecast 30 days
You will need to create an account on openweathermap.org then only you can use the APIs. You can choose as per your requirement and go for it. I have used the free API for this tutorial. You can simply signup with your email and get API key which is will be used to get the details.
You can chose the free tier if you have <1m calls per month. Please note, the Free Tier only supports a few weather services for forecasts. You can click on Get API Key on Free Tier.You can then enter your user details to get the API keyYou will have a final question on usage which looks like more for statistical information rather than anything else
Once created, you can then go to the menu: My API Keys from the user menu.
Once in the API Key area, you can use the default key, or generate a new key (if you’re generating a new key, the key name is not important).
Now that you have the key, you can then call the API to test if you can get the data for a given location.
Step 2: Testing Weather Forecast using Lat Long
In order to get the weather for a given location, you can use the Latitude (Lat) and the Longitude (Long) of a given location. The easiest way to get the Lat Long is to open up google maps, then search for a given location, then right click to get the Lat Long. The first number is a Latitude (e.g. in screenshot below 22.30130 is the Latitude)
Once you’ve got the Lat Long, then you can make a request to the API. The API we will be using is called “OneCall”. You can find the documentation here: https://openweathermap.org/api/one-call-api
Here’s a code snippet of code by passing in the Lat Long:
#weather_forecast.py
import requests
OPEN_WEATHER_MAP_APIKEY = '444ea47a72e5390ca5e538dca754b606'
def get_weather_data_by_location( lat, long):
url = f'https://api.openweathermap.org/data/2.5/onecall?lat={lat}&lon={long}&appid={OPEN_WEATHER_MAP_APIKEY}&units=metric'
print(f"Getting data via {url}")
r = requests.get(url)
return r.json()
if r.status_code == 200:
return r.json()
else:
return None
if __name__ == '__main__':
print("Getting Weather Data")
print( get_weather_data_by_location( '22.300910042194783', '114.17070449064359') )
And here’s the output:
As you can see, there is a LOT of data that comes out. We’ll make a slight update in order to just get the right fields. To make it more readable you can use an online JSON viewer by first running the program again, but then outputting the data to a file:
python3 weather_forecast.py > output_file.json
This will output all the data to a file called “output_file.json”. You can then open the file and copy and paste the data to Online JSON viewer like so:
You can then go to the “Viewer” tab to see formatted json.
The daily element contains 8 days of data. Records [0] is the current day at <current date> 4AM (GMT timezone). Records [1] is the <next day> 4AM (GMT timezone). Record [7] is the <7th day> 4AM (GMT timezone).
The field “dt” is the datetime field in unix epoch time (epoch time is the number of seconds since 1st Jan 1970). You can use a quick converter from epoch time at https://www.epochconverter.com
Hence, the code snippet to print just this data will be as follows:
##.. as per previous code above
if __name__ == '__main__':
print("Getting Weather Data")
json_data = get_weather_data_by_location( '22.300910042194783', '114.17070449064359')
for index in range( 1, 8):
print( f" Day [+{index}] {json_data['daily'][index]['dt']} = { json_data['daily'][index]['temp']['day']} " )
Output will be as follows:
Next, let’s convert the epoch time to local time. To do this, we will use the timezone module pytz and the datetime function. Please note you have to include the libraries datetime and pytz.
import datetime, pytz
##.. as per previous code above
if __name__ == '__main__':
print("Getting Weather Data")
json_data = get_weather_data_by_location( '22.300910042194783', '114.17070449064359')
for index in range( 1, 8):
local_time = datetime.datetime.fromtimestamp( json_data['daily'][index]['dt'] , tz=pytz.timezone('Asia/Singapore'))
str_time = local_time.strftime( '%Y-%m-%d %a' )
print( f" Day [+{index}] {str_time} = { json_data['daily'][index]['temp']['day']} " )
In the above code, we call “datetime.datetime.fromtimestamp” and pass in the epoch time, and a local timezone you want to convert from. You can find all the list of valid timezones on wikipedia timezone list and chose your own timezone.
The “local_time.strftime” will then format the time. We are just getting the Year-Month-Date DayOfWeek.
Output as follows:
Let’s finally view this as a graph online.
Step 4: View the daily forecast on a graph online
In order to show this as a bar chart, we are going to use an online service called Quick Chart which is developed by Ian Webster. The service helps to generate a chart online via a query link or to download an image file.
First install the quickchart library using pip or pip3 (depends on your installation):
pip3 install quickchart.io
Then from the imported class QuickChart, you can assign the parameters to draw the chart via the field .config. Once that’s done, you can get a URL and a downloaded image.
The following helper function takes in the same dictionary from the original json data from the openweathermap website, and then inserts it into the chart.
from quickchart import QuickChart
#as per previous code...
def get_quick_chart( json_data , output_file):
qc = QuickChart()
qc.width = 500 #set width and height of chart in pixels
qc.width = 500
labels = [] #Declare to hold the x-axis tick labels
weather_readings = [] #get the data labels#Just as per before, convert date, and get the forecast days
for index in range( 1, 8):
local_time = datetime.datetime.fromtimestamp(json_data['daily'][index]['dt'] , tz=pytz.timezone('Asia/Singapore'))
labels.append( local_time.strftime( '%a %d/%m ' ) )
weather_readings.append( round( json_data['daily'][index]['temp']['day'] ,1) )
qc.config = """{ type: 'line',
data: { labels: """ + str( labels ) + """,
datasets: [ {
backgroundColor: 'rgb(255, 99, 132)',
data: """ + str( weather_readings) + """,
lineTension: 0.4,
fill: false,
}
],
},
options: {
title: { display: true, text: '7-Day Weather Forecast' },
legend: { display: false},
scales: { yAxes: [ { scaleLabel:
{ display: true,
labelString: 'Temperature Degrees Celcius' } } ]},
plugins: {
datalabels: {
display: true,
align: 'bottom',
backgroundColor: '#ccc',
borderRadius: 3
},
}
},
}"""
print(qc.get_short_url()) #Print out the chart URL
qc.to_file(output_file) #Save to a file
With the above, you will get a URL with the chart, and also an image which is downloaded to the current working directory as follows:
Conclusions and full code
So we managed to get a weather forecast, and then also show this on a chart using QuickCharts.
You can see the full code from the GitHub repo from here
Happy Coding!
Get Notified of New Posts
We post regularly here. Get notifications of when new posts are available directly into your inbox and learn new tips and tricks you would only stumble across when you really are stuck!
Six minutes of coding and 6 hours debugging. Well, it is inevitable that most of your time will be spent on debugging rather than coding in your early stage of programming. Debugging is often annoying, frustrating, and unpredictable to know how much time it takes. In order to have a clean code while debugging would be great.
While debugging one needs to have the skill to read the code and tracebacks. Learning about debugging in the early days of programming can be really beneficial. In this tutorial, I am going to talk about how to debug your code the smarter way in python 3.8 and above and you can save a huge amount of time.
Introduction to debugging
Debugging is one of the most important parts of software development. It is the process of finding incorrect operation, calculation of steps called bugs, and the process of analysing these bugs is called debugging.
There can be multiple reasons for the failure of your code and debugging can be really painful at first. With experience in coding, and using the correct tools and modules, we can find our bug quickly and solve it. It is a really important skill that every developer should have.
In the early days of my own programming, I was using print statements to debug my code. At a later stage, I was introduced to the pdb standard library of Python. There is no need to install extra packages. Although for a smaller program we can use just a print statement, but as your code gets bigger with different modules and different classes, this is not so simple.
Some of the drawbacks of using print() include:
For large code with lots of modules, functions, and classes, adding the print() statement is quite painful
The code is not interactive and hence will execute automatically without any chance to investigate specific lines of code
Once all the debugging is complete, we still have to delete those print statements
How to use the python breakpoint()
Starting in Python 3.7, another efficient and flexible built-in function was introduced called breakpoint() which was easy to remember. In earlier versions before 3.7 , you can achieve a similar outcome by importing a pdb module and calling the pdb.set_trace().
It was very useful in the early phase as by inserting only the import pdb;pdb.set_trace() we are able to pause our execution and python will bring up the interactive debugger. You can then enter a command and see all the code executed and access variables.
Simple Divide Function
Let’s start the scenario without using breakpoint() in the simple example given above. We have created the function divide which will take two parameters numerator, denominator and calculate the division operation. It will work if we simply put correct arguments that can be divisible but if we put 0 on the denominator as above, we get the following exception:
Getting Zero Division Error
As you can see the tracebacks and exceptions, we have ZeroDivisionError which is very obvious since the code is fairly simple. But imagine if we have thousands of lines of code with many variables, well this is the point where our breakpoint() comes in handy.
By simply adding the breakpoint() above the line where an error has been raised, we can see what is being processed till now. By checking the variable value, we can get to know what is actually causing the error by doing operation in pdb console.
Now let’s inject breakpoint and execute the function.
using breakpoint()
As we saw the error was happening on return operation so we put the breakpoint just above it.
By simply running the script we can see the pdb shell. You can also change the value of variables during runtime to test your hypotheses to know what is the reason behind the failure of the code.
After correcting the variable of denominator we can use the command word c which will use to continue and execute the remaining process. Another handy command is q to quit from the console. Another common set of commands include n which will execute the next line (but step over any functions). And finally, the s command will step into a function
Trying to find what is the errror
After checking what is coming in the numerator and denominator and using the value we found while dividing by zero, we get an error. This was possible only breaking up our code flow and pausing the code in the required field using breakpoint().
Now to solve this issue we can change the code in a different way. My approach would be printing the message that we can’t perform the process if the denominator is zero and execute the process without error.
Now using knowing what was causing error we can write the code bug-free.
So simply adding the breakpoint in the required line and executing the function. All commands can be found using h on console
This will give us all the commands
Some of the useful commands that we may use frequently while debugging are :
c (continue) will continue to execute the code after breakpoint till the point where another breakpoint is defined
n will execute the next line of code
s will step into the function if present and will execute the function,if not just as as
p can be used to print to test the variable value and class,q will quit the debugger/execution,
h if you need help with debugger use ‘h’(help), which lists all the options
args list out the arguments in the current function
One thing to be careful is not to use any of the command line names as variable names in your code. I spent quite some time trying to see why my variable “args” does not have the expected value only to realise that args shows me the current arguments.
Tips and Tricks With Python Debuggin
Here are some tips that will certainly save you a bunch of time when using the debugger:
Add the “breakpoint()” function inside an IF statement to only fire when the condition you’re trying to debug comes up. It’ll help you find issues faster
Use dir( object ) in order to get a list of all the available elements and functions that a given object has
Use object.__dict__ to convert the current object into a dictionary to list our all element names and values. You can get a glimpse of an object very quickly
Run your flask server in the foreground and use breakpoint() in your flask apps so that when you perform a web function, your breakpoints will fire
Conclusion
The breakpoint is a powerful weapon to debug Pythonic code which adds “effectiveness” as there’s no mess of print() statements in your code and makes your clean much cleaner and effective.
In this tutorial, we are going to learn about how to automate the process of removing files in a specific folder using python. We are going to use different packages but mainly our programming logic will help us.
No matter which kind of OS you are using, Windows, Mac, or Linux, you always will be worried about space. Nowadays 1 TB hard disk looks smaller. Talking about myself, I am a windows user and I have 128 GB SSD as a C drive. But I can barely use 25 GB. So, there might be some unnecessary files that I download throughout the week, and if we clean them then we might get rid of useless files. Though, this task seems boring. Yes, none of us want to do it. It takes time as we need to check the date on each one and them select and remove it.
That’s where our python comes to the picture. Python is a very useful language to automate this kind of boring stuff. What we need the right mindset, python IDE and some time to write and setup script. So, in this tutorial, we will try to write a script that removes the files which are older than a week from the specified path and also put it into output logs.
Let’s Get The General Idea First
The basic approach is we are going to enter the path and python will give us a file list in that directory. Then, we are going to check if the file is more than 7 days older. if yes then delete and put the entry in the log file. But, how we will get a file list? We can fetch that using os package. Os package gives us basic functionality which we can see by the right click of the mouse such as copy, rename, move, delete, etc.
Let’s start working on the script
import os
import datetime
import glob
path = 'enter_path_here'
today = datetime.datetime.today()#gets current time
os.chdir(path) #changing path to current path(same as cd command)
#we are taking current folder, directory and files
#separetly using os.walk function
for root,directories,files in os.walk(path,topdown=False):
for name in files:
#this is the last modified time
t = os.stat(os.path.join(root, name))[8]
filetime = datetime.datetime.fromtimestamp(t) - today
#checking if file is more than 7 days old
#or not if yes then remove them
if filetime.days <= -7:
print(os.path.join(root, name), filetime.days)
os.remove(os.path.join(root, name))
First, we are going to remove files without worrying about logging. Then, we will write details in log files. Here, we need two libraries First one is os as mentioned, and the second one is datetime. After importing we stored the path and current date in the variable. We used the os.walk() function for traversing around the file tree. os.walk() gives also a file that is under sub-directory and we surely want to remove that.
So, in that loop, we check every file using the os.stat() function. os.stat provides us basic info/metadata about the file such as last modification time, permission, and so on. So, by using that, we stored the last modification time and stored it into the variable. But there is a small problem, we got that value in seconds and we need to convert it into the date. That’s when datetime package will help. Finally, we compare if the difference between both of them more than 7 days then we print the full file path, the difference in days, and remove that file using the os.remove() function.
Let’s Make a Log File
This is typically an operation you run automatically, but things may go wrong or you may want to audit the files you have deleted. This is where keeping a track of the deletions in a log file will help. For making a log file, you just need to know the basics of file handling using python and you can do it by yourself. Let me help you with that. We will modify our script with a few small changes by creaing a simple log file.
import os
import datetime
import glob
path = #'enter your path here'
logging_path= #'Enter your log directory path here'
#making a log directory at given path(if exists then it will skip)
if not os.path.isdir(logging_path):
os.mkdir(logging_path)
else:
print("Directory already exists")
today = datetime.datetime.today()
os.chdir(path)
#creating a log file with date
file=open(logging_path+datetime.datetime.today().strftime('%d-%m-%Y')+'.txt','a')
for root,directories,files in os.walk(path,topdown=False):
for name in files:
t = os.stat(os.path.join(root, name))[8]
filetime = datetime.datetime.fromtimestamp(t) - today
if filetime.days <= -7:
print(os.path.join(root, name), filetime.days)
file.write(os.path.join(root, name)+' created '+str(-1*filetime.days)+' days ago\n') #writing in the file
os.remove(os.path.join(root, name))
First, we need to create a log directory. If that directory is available then we print Directory is already available. then we create a text file with today’s date as a name in append mode. Then we write in that file File full path and name and how many days before that file was created. That is enough about logging. By using this simple technique, you can check earlier deleted files by the same script.
Making your file deletion python script dynamic
Yes, we automated the deleted process but it kind of boring if you have to change path in the script every time. So, what we are going to do now is you can run files from outside and add a folder path in argument and our code will take care of all other things. Let’s do that now.
import os
import datetime
import glob
import sys
path = sys.argv[1] #argv[0] is your current python file and argv[1] is first argument where you want to delete files from
logging_path='Enter your log directory path'
if not os.path.isdir(logging_path):
os.mkdir(logging_path)
else:
print("Directory already exists")
today = datetime.datetime.today()
os.chdir(path)
file=open(logging_path+datetime.datetime.today().strftime('%d-%m-%Y')+'.txt','a')
for root,directories,files in os.walk(path,topdown=False):
for name in files:
t = os.stat(os.path.join(root, name))[8]
filetime = datetime.datetime.fromtimestamp(t) - today
if filetime.days <= -7:
print(os.path.join(root, name), filetime.days)
file.write(os.path.join(root, name)+' created '+str(-1*filetime.days)+' days ago\n')
os.remove(os.path.join(root, name))
Here, we first imported the sys package. sys package is a useful package for managing command-line arguments. Here, the filename is the first argument and our path is the second argument. Thus we set argv[1] as our path. After that, our whole code is the same. Remember, don’t use that code unnecessarily. Otherwise, it will delete your files.
That’s an easy way to delete files. Let me show you my outputs.
I removed unnecessary temp filesLog has also the same details
Wooh…we are done with the coding part. But you have the correct question in mind. Do we need to run this script manually? The answer is no. We can schedule this task using a task scheduler in windows.
Automate Code Using Task Scheduler on Windows
Search task scheduler
First, you need to open the task scheduler. Then on the right side, you can see create basic task options. After clicking on that new window will open.
create basic task screen
No, you need to enter name and description. Then go to the next screen
choose daily
After this, you have to give a starting date and time. Then on the next screen choose to start a program. On the final screen choose program path and argument as to which folder you want to clean.
Choose path and argument
After following every step perfectly, your script will run automatically and you have absolutely automate whole boring stuff using python.
