For most serious applications, you will often have to have persistent storage (storage that still exists after your applications stops running) of some sort. For new developers, it can be quite daunting to decide which option to go for. Is a simple flat file enough? When should you use something like a database? Which database should you use? There are so many options that are available it becomes quite daunting to decide which way to go for.
This is a starting guide to provide an overview of some of the many data storage options that are available for you and how you can go about deciding. One thing to keep in mind is that if you are developing an application which is either planned or has a possibility to scale over time, your underlying database might also grow overtime. It may be quick and easy to implement a file as storage, but as your data grows it might be better to use a relational database but it will take a little bit more effort. Let’s look at this a bit deeper
What are the possible ways to store data?
There are many methods of persistent storage that you can use (persistent storage means that after your program is finished running your data is not lost). The typical ways you can do this is either by using a file which you save data to, or by using the python pickle mechanism. Firstly I will explain what some of the persistent storage options are:
File: This is where you store the data in a text based file in format such as CSV (comma separated values), JSON, and others
Python Pickle: A python pickle is a mechanism where you can save a data structure directly to a file, and then you can retrieve the data directly from the file next time you run your program. You can do this with a library called “pickle”
Config files: config files are similar to File and Python Pickle in that the data is stored in a file format but is intended to be directly edited by a user
Database SQLite: this is a database where you can run queries to search for data, but the data is stored in a file
Database Postgres (or other SQL based database): this is a database service where there’s another program that you run to manage the database, and you call functions (or SQL queries) on the database service to get the data back in an efficient manner. SQL based databases are great for structured data – e.g. table-like/excel-like data. You would search for data by category fields as an example
Key-value database (e.g redis is one of the most famous): A key-value database is exactly that, it contains a database where you search by a key, and then it returns a value. This value can be a single value or it can be a set of fields that are associated with that value. A common use of a key-value database is for hash-based data. Meaning that you have a specific key that you want to search for, and then you get all the related fields associated with that key – much like a dictionary in python, but the benefit being its in a persistent storage
Graph Database (e.g. Neo4J): A graph database stores data which is built to navigate relationships. This is something that is rather cumbersome to do in a relational database where you need to have many intermediary tables but becomes trivial with GraphQL language
Text Search (e.g. Elastic Search): A purpose built database for text search which is extremely fast when searching for strings or long text
Time series database (e.g. influx): For IoT data where each record is stored with a timestamp key and you need to do queries in time blocks, time series databases are ideal. You can do common operations such as to aggregate, search, slice data through specific query operations
NOSQL document database (e.g. mongodb, couchdb): this is a database that also runs as a separate service but is specifically for “unstructured data” (non-table like data) such as text, images where you search for records in a free form way such as by text strings.
There is no one persistent storage mechanism that fits all, it really depends on your purpose (or “use case”) to determine which database works best for you as there are pros and cons for each.
Optional. You can create a config file before hand
Yes – you can use any text based editor
Small
Slow
Slow
No – manual
Database SQLite
None – database created automatically
No – only in python
Small-Med
Slow-Med
Slow-Med
No – manual
Relational SQL Database
Separate installation of server
Through the SQL console or other SQL clients
Large
Fast
Fast
Yes, require extra setup
NoSQL Column Database
Separate installation of server
Yes, through external client
Very large
Very fast
Very fast
Yes, inbuilt
Key-Value database
Separate installation of server
Yes, through external client
Very large
Very fast
Fast-Very Fast
Yes, require extra setup
Graph Database
Separate installation of serverSeparate installation of server
Yes, through external client
Large
Med
Med
Yes, require extra setup
Time Series Database
Separate installation of server
Yes, through external client
Very large
Very fast
Fast
Yes, require extra setup
Text Search Database
Separate installation of server
Yes, through external client
Very large
Very fast
Fast
Yes, require extra setup
NoSQL Documet DB
Separate installation of server
Yes, through external client
Very large
Very fast
Fast
Yes, require extra setup
A big disclaimer here, for some of the responses, the more accurate answer is “it depends”. For example, for redundancy for relational databases, some have it inbuilt such as Oracle RAC enterprise databases and for others you can set up redundancy where you could have an infrastructure solution. However, to provide a simpler guidance, I’ve made this a bit more prescriptive. If you would like to dive deeper, then please don’t rely purely on the table above! Look into the documentation of the particular database product you are considering or reach out to me and I’m happy to provide some advice.
Summary
There are in fact plenty of SaaS-based options for database or persistent storage that are popping up which is exciting. These newer SaaS options (for example, firebase, restdb.io, anvil.works etc) are great in that they save you time on the heavy lifting, but then there may be times you still want to manage your own database. This may be because you want to keep your data yourself, or simply because you want to save costs as you already have an environment either on your own laptop, or you’re paying a fixed price for a virtual machine. Hence, managing your own persistent storage may be more cost effective rather than paying for another SaaS. However, certainly don’t discount the SaaS options altogether, as they will at least help you with things like backups, security updates etc for you.
Python is among the top programming languages that have been used in recent years in designing high-end technologies, such as Machine Learning, artificial intelligence, and data science. Programmers also use Python as their language of choice in developing large-scale applications that scale several products and services. This is why reputed companies hire candidates with good knowledge in coding with Python and other programming skills.
However, despite all these, some python myths can be a concern for aspiring developers. Below are some of the python programming myths you can easily come across.
1. Python is Slow
While Python is admissibly slower than Java and C++, it responds faster than JavaScript, Ruby, and other languages. Python features have specific runtimes and are not slower than other languages. Therefore, using Python for complicated applications saves time, and you’ll be done in a few minutes.
Some years ago, CPUs and memory were costly. However, currently, you can buy better-performing hardware at an affordable price to support programming with Python. Python also supports several programming paradigms, making it functional and imperative.
2. Python is Not Compiled and Only Used for Scripting
Python is generally an interpreted coding language since it falls in this category but is also considered a compiled language like Java and other programming languages. The compiling process is automated, making it difficult to detect, and a separate compiler isn’t required. It mostly compiles on virtual machines.
Python isn’t a scripting language wholly but more of a general-purpose coding language that can be used for scripting. Like most scripting languages, Python doesn’t have networking, regular expression, and exception features. This makes it a reliable and trusted programming language that can automate several tasks.
3. Learning to Code with Python is difficult and Time-consuming
Learning to program with Python is easy as it doesn’t require any prior programming knowledge. However, coding experts are advantaged as they can easily relate to its concepts. Python is a high-level language that can easily be implemented. Most of its syntax is simple mathematical instructions and calculations.
Most statements written in python programs look familiar with the English language as it contains less syntax. That said, learning to code with Python can take between three to six months, depending on your commitment. Besides, there are plenty of learning resources and a large supporting community that is ready to help learners.
4. Python is Not Scalable
Contrary to what most people believe, Python can scale both horizontally and vertically better than other languages. However, there is some confusion about this. The scaling process isn’t automated, thus requires some engineering effort. Scaling Python isn’t a straightforward process as it requires several entities.
For instance, you should make the most from the underlying memory, enhance single systems into distributed form, and more. Nonetheless, with proper architecture, scaling Python won’t be a problem.
5. Coding with Python is Expensive
You are highly mistaken if you think python programming is expensive. Unlike other coding languages, Python is an open-source language that can be downloaded for free from its official website. Python was officially developed in 1991 and is managed under the Python Software Foundation, which guarantees small and large scale users an Open Source License.
However, most of Python’s licenses remain open-source, though others are not. Some contributions, especially those from the General Public License, require users to pay a fee to access customizations added by other developers.
6. Python has Support and Security Issues
Another common myth is that Python isn’t secure, and code lines can easily be hacked. Most programmers believe the assumption that python codes are prone to cyberattacks. In contrast, Python has been used to build networking security systems. The language is also used to develop security testing tools and automation testing, which perform faster compared to others.
On the other hand, Python’s support team is always on standby and ready to assist in case of security issues affecting python programmers. You can contact them anytime, and your details will be kept confidential. Python has also adapted PayPal, eBay, and other highly-secured third-party payment gateways to prove its legitimacy.
7. Python Cannot be used for Big Projects
Just because Python is a simple language doesn’t mean it cannot be applied in big projects. Python has reusable codes and an extensive predefined library, which allow developers to create new codes tailored to suit project needs. Python libraries are also reusable, reducing the amount of time and effort required to write codes. Other languages are quite complicated and take long before a program is designed and implemented. This explains why tech giants, including Google, Facebook, YouTube, and Instagram, use this language.
Major websites / internet services written in Python
The Bottom Line
There is a lot to discover about Python and other programming languages in general. You shouldn’t agree easily to some of these baseless and unproven myths and misconceptions, which often arise during specific situations. That said, if you have some python programming basics, check out this course to learn UX/UI design and advance your skills to also expand your applications to cover front end as well.
If you are new to the world of computer programming, choosing a programming language, to begin with, is probably the toughest hurdle. Currently, there are thousands of programming languages with different idiosyncrasies and complexities. On our site, we focus on Python, but there are other languages out there. Before you start your software development journey, choosing a programming language that suits your interests and career goals is important. That said, below are some of the best and in-demand coding languages you should consider.
1. JavaScript
Modern software developers cannot succeed without mastering JavaScript. A 2020 survey done by Stack Overflow found that JavaScript is still the most popular programming language for developers for eight years in a row. More than 70% of study participants reported that they used this language for more than one year.
Together with CSS and HTML, JavaScript is an important coding language for front-end website development. Most websites, including Facebook, Gmail, YouTube, and Twitter, depend on JavaScript to display dynamic content to users for their interactive website pages.
Even though JavaScript is primarily a front-end web development language on browsers, it can be used on the server-side to develop scalable network applications with the help of Node.js. Node.js works with Windows, Linux, Mac OS, and SunOs.
JavaScript is a popular language amongst programming beginners because of its simple learning curve. It is used all through the web, thanks to its speed, and works well with other coding languages, enabling it to be used in various applications. That aside, the demand for JavaScript developers is currently high, with a CareerFoundry study concluding that 72% of businesses need JavaScript developers.
Pros of learning JavaScript
Fast and can run immediately in browsers
Provides an enriched and better web interface
Highly versatile
It can be used in various applications
Has multiple add-ons
Easily integrates with other programming languages.
Cons of learning JavaScript
Lacks an equivalent or alternate method
Different web browsers can interpret code lines differently.
2. Python
Python is a general-purpose coding language that is also very learner-friendly; there are even Python classes for children. However, despite being easy to learn, Python is an overly versatile and powerful language, making it suitable for beginners and experts. It is because of this that major companies, including Facebook and Google, use this language.
Python’s popularity is largely attributed to its extensive usage. It has applications in data science, scientific computing, data analytics, animation, database interfacing, web applications, machine learning, and data visualization. This versatility also explains the high demand for experts in this language.
Key features of Python include;
It has a unique selling point – simple, productive, elegant, and powerful in one package.
It influences other programming languages, such as Go and Julia
Best for back-end web development with first-class integration with other programming languages, such as C++ and C.
It offers many tools that can be applied in computational science, mathematics, statistics, and various libraries and frameworks, such as NumPy, Scikit-Learn, and Pandas.
Pros of learning Python
Works in various platforms
Improves developers and programmers productivity
Has a wide array of support frameworks and libraries
Powered by object-oriented programming
Cons of learning Python
Not ideal for mobile computing
It has a primitive and underdeveloped database
3. Java
Java is another popular coding language commonly used in-app and web development. Despite being an old coding language, Java is still in demand due to its complexity. Unfortunately, it isn’t beginner-friendly. It is a platform-independent language and a popular choice for various organizations, including Google and Airbnb, for its stability.
Key features of Java include;
It is a multi-paradigm and feature-rich programming language
Very productive for developers
Moderate learning curve
It doesn’t have major changes and updates like Python and Scala
Has the best runtime
Pros of learning Java
Has a wide array of open-source libraries
Automated garbage collection
Allows for platform independence
Supports multithreading and distributed computing
Has multiple APIs that support completion of various tasks, such as database connection, networking, and XML parsing
Cons of learning Java
Expensive memory management
Slow compared to other coding languages, such as C and C++
4. C#
C# is an object-oriented programming language developed by Microsoft. It was initially designed as part of the .NET framework for developing windows applications but is currently used in various applications. It is a general-purpose coding language used particularly in back-end development, game creation, mobile app development, and more. Despite being a Windows-specific language, it can also be used in Android, Linux, and iOS platforms.
The language has a legion of libraries and frameworks that have accrued for the last 20 years. Like Java, C# is independent of other platforms, thanks to its Common Language Runtime feature.
Pros of learning C#
Can work with shared codebases
Safe compared C++ and C
Uses similar syntax with C++ and other C-derived languages
Has rich data types and library
Has a fast compilation and execution
Cons of learning C#
Less flexible compared to C++
You should have good knowledge to solve errors
5. PHP
PHP is another excellent programming language with many applications. While it faces stiff competition from other languages, such as Python and JavaScript, especially for web development, there is still a high demand for PHP professionals in the current job market. PHP is also a general-purpose and dynamic coding language that can be used to develop server-side applications.
Pros of learning PHP
Easy to learn and use
Has a wide ecosystem and community support
Has many frameworks
Supports object-oriented and functional paradigms
Supports various automation tools
Cons of PHP
Builds slow web pages
Lacks error and security handling features
6. Angular
Angular is a recently updated and improved version of the initial AngularJS framework developed by Google. Compared to other recent coding languages, such as React, Angular has a steep learning curve but offers better practical solutions for front-end development. Developers can also program complicated and scalable applications using Angular, thanks to its great functionality, aesthetic visual designs, and business logic.
Key features of Angular include;
Features a model-view control architecture that facilitates dynamic modeling
Uses HTML coding language to develop user interfaces that are simple and easy to understand
Uses old JavaScript objects, which are self-sufficient and very functional
Has Angular filters, which filter data before being viewed
Pros of learning Angular
Requires minimal coding experience to use
Allows development of high-quality hybrid apps
Has quick app prototyping
Has enhanced testing ability
Cons of Angular
Angular developed apps are dynamic, diminishing their performance
Complicated pages in apps can cause glitches
Difficult to learn
7. React
Also called ReactJS, React is a JavaScript framework developed by Facebook that enables programmers to develop user interfaces with dynamic abilities. Sites built using React respond faster, and developers can switch between multiple variable elements seamlessly. The language also enables businesses to build and maintain customer loyalty by providing a great user experience.
Pros of learning React
Easy to learn and SEO friendly
Reuses various components, thus saves time
Has an open-source library
Supported by a strong online community
Has plenty of helpful development tools
Cons of React
Additional SEO hurdle
Has poor code documentation
The Bottom Line
As you choose your preferred web development language to learn, ensure that you aren’t guided by flashy inclinations and popularity contests. Even though the realm of computer programming keeps changing rapidly, the languages mentioned above can withstand these changes. Learning one or more of these languages will put you in a great position for many years to come. Make use of federal funding to pay for your online programming courses and Bootcamps. Veterans can learn web development languages at a discount using the GI Bill Benefits.
The easiest and simplest mechanism to store data from python is the humble file storage which is often, but does not have to be, text based. There are no libraries that you require, and you can use native python functions to open and write to the file very easily.
There are many use cases for file storage and is usually the “go to” method when hacking a quick solution or prototype together. These are also arguably good solutions for production use cases.
Overview of using storing data to files in Python
The typical use cases has the following commonalities:
Setup: There’s no setup that is required for files. You can create the file even from python
Volume: Size Small-ish file size (< 5-10mb). You can go larger of course if your application is not doing heavy reads or writes nor if it doesn’t require fast response (e.g. batch processing)
Record access: Does not require to search data within the file to extract just portion of the records. You would load or save all the data in the file in one go
Data Writes: You can either append to the file or you can upload and download all data in the file.
Write reliability: You do not need to have multiple writes at the same time – there is only possibility (or likelihood) of one person writing at one time, and if there was a case of multiple people writing at once, the consequence are not serious for your application. There are ways to put a lock on a file to prevent conflicts, but you should double check if a file is the write option for you
Data formats: You may have structured record based (such as comma separated value – CSV or tab delimited) or unstructured (eg document of text or JSON format). You can also store binary data in a file as well – e.g. for images
Editability: You may want or allow direct editing of the file by other applications or direct editing by people
Redundancy: There’s no inbuilt redundancy. If there is any failure (data corrupt, the server with the file fails), then you’re out of luck. You need to setup your own mechanisms (e.g. replicate file to another server automatically)
Code examples to read and write to a file
Here are two sets of example code for writing and reading from a file. It is very easy and does not require any libraries. The one thing to be mindful of is what mode you want the file to be opened- read, write, read and write.
Open a text file for (over)writing:
To write to a file, it’s very easy to do so which is to use the ‘w’ switch on the open() function. There are other options as well:
‘r’ – Reading
‘w’ – Writing to a file
‘a’ – Append to end of file
‘r+’ – Read and write to the same file
‘x’ – Used to create and write to a new file
file = open( ‘population.txt’, ‘w’)
file.write(‘Japan’)
file.write(‘United States’)
file.write(‘Australia’)
file.write(‘China’)
file.close() #file is released and closed
You will then have the following output file of population.txt:
Japan
United States
Australia
China
Open a text file fully for reading:
Using the same population.txt file created above –
file = open( ‘population.txt’, ‘r’)
data = file.read() #read full contents of file into a single string
file.close() #file is released and closed
print(“*** file start ***”)
print( data )
print(“*** end file ***”)
The output would be:
*** file start ***
Japan
United States
Australia
China
*** end file ***
Now to explain this a bit further, the open() command helps to open a file where you need to specify how the file is to be opened – in this case with ‘r’ to indicate it is for reading. There are other options as well:
‘r’ – Reading
‘w’ – Writing to a file
‘a’ – Append to end of file
‘r+’ – Read and write to the same file
‘x’ – Used to create and write to a new file
Read a text file line by line:
file = open( ‘population.txt’, ‘r’)
data_list = file.readlines() #read full contents of file into a list of rows
file.close() #file is released and closed
print(“*** file start ***”)
counter = 0
for row in data_list:
counter = counter + 1
print( f”{counter}: {data_list}” )
print(“*** end file ***”)
The output would be:
*** file start ***
1: Japan
2: United States
3: Australia
4: China
*** end file ***
The difference in above to the first example is that the data comes out in a list separated by a newline so that you can process each row. Please note, you can simplify the above using the enumerate to avoid having the separate counter variable setup. E.g.
print(“*** file start ***”)
for index, row in enumerate(data_list):
print( f”{index+1}: {data_list}” ) #Note that when using enumerate, first index is 0
print(“*** end file ***”)
Summary of writing and reading to a file
Reading and writing to a file is a very straightforward native operation in Python. There are many other related operations that you can do ranging from putting a lock on a file to prevent two processes writing to the same file, checking file attributes such as access and size, and many other operations. At the most basic though, you can simply use the “open” statement to do the read/write to satisfy most of your needs.
A config file is a flat file but is used for reading and writing of settings that affect the behaviour of your application. These files can be incredibly useful so that you can put individual settings inside the human editable file and then have the settings read from your application. This helps you configure your application in the way you need without having to change the application code.
Typically the config file is edited by a simple text editor by the user, then the application runs and reads the config file. If there are any changes to the config file, normally (depending how the code is written), the application will then have to be restarted to take on the new settings.
Some of the considerations for using a config file as a “data store” includes:
Setup: There’s no setup that is required for files. You should use one of the config management python libraries that are available to make it easier to manipulate config files.
Volume: Size Small-ish file size (< 5-10mb)
Record access: Does not require to search data within the file to extract just a portion of the records. You would load or save all the data in the file in one go
Data Writes: Applications don’t generally write to a config file, but it can be done. Instead the config file is edited outside in a text editor
Data formats: Normally the data would be a structured record based (such as comma separated value – CSV or tab delimited), or a more complex structure such as what you see in windows based .INI files or JSON format even
Editability: You generally want to allow direct editing of the file by users
Redundancy: There’s no inbuilt redundancy. If there is any failure (data corrupt, the server with the file fails), then you’re out of luck. You need to setup your own mechanisms (e.g. replicate file to another server automatically)
Code examples to read and write from config file using ConfigParse
Setting up a config file is actually not that much harder than simply creating a constants inside your application. Your main decision will be what type of configuration file format you’d like to use as there are quite a few to choose from. Here are some options and samples:
Example 1: Simple text file which is tab-delimited
You can see a full article on how to read a text file in our “Storing Data in Files in Python” article. The short version of open a tab delimited file is as follows:
Suppose you have a configuration file as follows where each row has two fields which is separated by a tab:
Some explanation may be required on the code though to make it easier to understand. Firstly, the for loop is used to read a record line by line. So each time the for loop iterates, it will read a line into the field rec until the whole file is read.
The following code is a little tricky, but the intent is to take the two columns in the tab delimited file and create a dictionary key value pair.
It first removes the newline character from the end of the line (through rec.strip() )
This will then return a string which is then split with split() by the a tab characters (denoted by ‘\t’)
The result of this is a two filed array which is then created into a tuple format
The tuple is then put in a list and added to list with the [] brackets
The dictionary .update() method is used to finally add they key value pair
Example 2: A properties file with key value pair
If you have a fairly simple configuration needs with just a key-value pair, then a properties type file would work for you where you have <config name> = <config value>. This can be easily loaded as a text file and then the key-value be loaded into a dictionary.
The following code could easily load this configuration:
config = {}
with open('config_data.txt', 'r') as file_hander:
for rec in file_hander:
if rec.startswith('#'): continue
key, value = rec.strip().split('=')
if key: config[key] = value
print( config )
Here the code ignores any comment lines (e.g. the line starts with a ‘#’), and then string-splits the line by the ‘=’ sign. This will then load the dictionary ‘config’
[default]
name = development
host = 192.168.1.1
port = 31
username = admin
password = admin
[database]
name = production
host = 144.101.1.1
You can then read the file with the following simple code:
import configparser
config = configparser.ConfigParser()
#Open the file again to try to read it
config.read('test.ini')
print( config['database'][‘name’] ) #This will output ‘production’
print( config['database'][‘port’] ) #This will output ‘31’. As there is no port under# database the default value will be extracted
Example 4: Reading Config values from a JSON file
With JSON being so popular, this is also another alternative you could use to keep all your config data in. It is very easy to also load.
Assume your config file is as follows: config_data.txt
A config file is a great option if you are looking to store settings for your applications. These are usually loaded at the start of the application and then can be loaded into a dictionary which can then serve as a set of constants which your application can use. This will both avoid the need to hardcode settings and also allow you to change the behaviour of your application without having to touch the code.
