What software is, how it is built, and the tools developers use every day — applications, libraries, frameworks, APIs, editors, terminals, packages, version control, and open source
"The best programs are written so that computing machines can perform them quickly and so that human beings can understand them clearly."
— Donald KnuthEvery day, people use software without thinking much about what software actually is, where it comes from, who builds it, or what tools its creators use. A student opens a web browser. An office worker launches a spreadsheet. A musician records tracks in a digital audio workstation. A scientist runs a data analysis script. All of these people are using software — but very few of them have thought carefully about what software is, how it is organized, and what the process of creating it looks like from the inside.
Chapter 17 answers those questions. In Chapter 16, you learned that an operating system manages hardware and creates a stable environment for programs to run. In this chapter, you will learn what runs inside that environment — applications, utilities, libraries, and frameworks — and how the people who build software do their work. You will explore the tools developers use every day: code editors, integrated development environments, terminals, package managers, and version control systems.
By the end of this chapter, you should understand the software ecosystem not just as a user but as someone who can read about it, discuss it, and begin to work within it. Whether you plan to become a developer, a data scientist, a systems administrator, or simply a more informed citizen of a software-driven world, the ideas in this chapter will serve you.
When a developer wants to build an app that shows nearby restaurants on a map, do they build their own mapping system from scratch? Do they write their own payment processing code? Do they design their own database engine? The answer to each question is almost certainly no. Modern software development is built on layers: libraries that provide reusable code, APIs that allow programs to communicate with specialized services, frameworks that provide a structure for building applications, and tools that manage all of it. Understanding these layers helps you understand how software actually gets built — and why the ecosystem of existing code that surrounds any new project is as important as the project itself.
When most people say "software," they mean the apps on their phone or the programs on their computer. That is a reasonable starting point, but software is broader than apps. Software is any set of instructions that a computing system can execute. Software tells the hardware what to do. Without software, a computer is an expensive collection of electronic components that does nothing at all.
Software takes many forms. The application you use to browse the web is software. So is the operating system managing your files and memory in the background. The compiler that transforms source code into machine instructions is software. The database engine storing millions of records is software. The firmware embedded in a car's brake system is software. The algorithm running on a remote server that decides which video to recommend to you next is software. All of these differ in what they do, who uses them, and how they are built — but they share the same fundamental nature: they are instructions that a machine follows.
A useful mental model is the software stack — the layers of software that sit on top of hardware and on top of each other. Each layer depends on the layer below it and provides services to the layer above it.
When you click a button in an app, that click triggers a chain of activity passing through multiple layers before anything changes in hardware. The application sends a request to a library. The library makes a call through the operating system. The operating system communicates with a hardware driver. The driver instructs a physical component. The result travels back up the same chain. This layered organization is one of the central ideas of computer systems — and it is the same principle of abstraction that you encountered in Chapter 13.
Each layer hides the complexity of the layers below it. An application developer does not need to know how the operating system manages memory; the operating system does not need to know how the CPU executes an instruction. This controlled hiding of detail is what makes large, complex software systems manageable to build.
An application — often called an "app" — is software designed for a particular task that a user wants to accomplish. Applications sit at the top of the software stack and are the most visible kind of software to most people.
| Category | Purpose | Examples |
|---|---|---|
| Productivity | Create, edit, and manage work | Word processors, spreadsheets, project management tools |
| Communication | Send messages, hold calls, share files | Email clients, chat apps, video conferencing |
| Entertainment | Games, media, streaming | Games, music players, streaming services |
| Creative | Design and produce media | Image editors, video editors, audio workstations |
| Educational | Learning and reference | Language apps, simulations, encyclopedias |
| Business | Manage operations and customers | Accounting software, inventory systems, CRM tools |
| Scientific | Research and analysis | Simulation tools, statistical packages, CAD software |
A utility is a small program that performs a specific support task rather than a user-facing creative or productive task. Utilities are often used by system administrators, developers, or power users who need to maintain, configure, or analyze a computing environment. Common utilities include file compression tools, disk space analyzers, system monitors showing CPU or memory usage, text editors for configuration files, antivirus scanners, backup tools, and disk repair programs.
The distinction between an application and a utility is not always perfectly sharp — a text editor might count as either, depending on how it is used. What matters is the underlying idea: some software is designed for end users doing productive work, and some software is designed to support, configure, and maintain the computing environment itself.
When developers write software, they do not start from nothing. Most real programs rely heavily on libraries — pre-written collections of code that solve common problems. A library is a collection of functions, classes, or modules that a program can import and call to perform specific tasks. Instead of writing everything from scratch, a developer imports a library and uses the functions it provides.
Consider what it would take to write a Python program that reads a spreadsheet, calculates some statistics, and draws a chart without any libraries. You would need to write code to parse the spreadsheet file format byte by byte, implement your own statistical formulas, and draw pixels to the screen. That could take months of work. With libraries, the same task becomes a few lines of code:
Three lines of meaningful work, made possible by libraries that represent years of engineering effort by other people. Libraries exist for almost every area of computing.
| Task | Common Library Examples |
|---|---|
| Data processing | pandas, NumPy |
| Web requests | requests, httpx |
| Image processing | Pillow, OpenCV |
| Machine learning | scikit-learn, TensorFlow |
| Working with dates | datetime, dateutil |
| Cryptography | cryptography, hashlib |
| Testing | unittest, pytest |
A framework is larger and more opinionated than a library. While a library gives you tools to use as you choose, a framework gives you a structure within which you build your program. The key difference is who calls whom: when you use a library, your code calls the library's functions. In a framework, the framework calls your code at defined points. You fill in the parts specific to your application; the framework handles everything else.
A library is like a toolbox. You pick up the tools you need and use them in your own way, on your own schedule.
A framework is like a building structure. The frame is already in place; you fill in the walls, rooms, and furnishings according to the framework's rules and expectations.
Web frameworks like Django and Flask (Python), Express (JavaScript), or Ruby on Rails provide a structure for building web applications. They handle the mechanics of receiving web requests, routing them to the right code, formatting responses, and managing database connections — so developers can focus on what their specific application does, rather than rebuilding those mechanics from scratch for every project. Testing frameworks like pytest or JUnit provide a structure for writing and running tests. Game frameworks provide a structure for building interactive graphical programs.
Understanding libraries and frameworks helps explain why software development is faster than it used to be, why a bug in a widely-used library can affect thousands of programs simultaneously, and why evaluating whether to trust and depend on a particular library is an important professional skill.
An API — Application Programming Interface — is a defined set of rules for how one piece of software can communicate with another. An API specifies what requests can be made, what format those requests should take, and what responses to expect back.
The word "interface" is the key. An interface is a defined point of contact between two things. An electrical outlet is an interface: it has a defined shape and voltage, so any compatible plug can use it without knowing anything about the wiring inside the wall. Similarly, an API defines how software can connect to a service without needing to understand how that service works internally.
Most APIs you will encounter in modern programming are web APIs — they work over the internet using the same HTTP protocol used by web browsers. A program sends a request to a URL (called an API endpoint), and the service sends back data in a structured format called JSON.
JSON (JavaScript Object Notation) is a common text format for structuring data in API responses. It is human-readable and straightforward to parse in most programming languages:
APIs make it possible to build complex software by combining specialized services. A ride-sharing app might use a maps API for route calculation, a payments API for processing fares, a messaging API for driver notifications, and a location API for position tracking — rather than building any of those capabilities from scratch. This approach of assembling capabilities from APIs is sometimes called composing services.
Understanding APIs helps explain why apps can share data across platforms, why changing one service can break multiple apps that rely on it, why API security matters enormously (APIs are where different systems share data), and why pricing changes at major APIs can affect entire ecosystems of apps overnight.
requests into their Python program. Where does this library come from, and what problem does importing it solve?{"user": "alice", "score": 92, "level": "advanced"}. What three pieces of information are in the response?Writing software requires tools. The most fundamental is something to write code in. Because code is plain text, any text editor can technically be used — but tools designed specifically for code offer features that make the work significantly faster, more accurate, and less error-prone.
A code editor is a text editor built for writing and reading code. It offers features that general word processors like Microsoft Word do not. The most visible feature is syntax highlighting, which colors different parts of code differently based on what they mean — keywords appear in one color, strings in another, numbers in another, and comments in yet another. This makes code dramatically easier to read at a glance.
Other key features of code editors include auto-indentation (new lines are automatically indented to match the surrounding code structure, which matters especially in Python where indentation carries meaning), line numbers (letting developers refer to specific locations when discussing or debugging code), and search and replace (letting developers find and change patterns across large files or entire projects). Popular code editors include Visual Studio Code, Sublime Text, and Notepad++.
An IDE — Integrated Development Environment — combines a code editor with a comprehensive set of development features in a single application. The key word is "integrated": instead of switching between separate tools, the developer has everything in one place.
| IDE Feature | What It Does |
|---|---|
| Code completion | Suggests how to finish a function or variable name as you type, reducing typos and speeding up writing |
| Error highlighting | Underlines errors in code as you type, catching many syntax problems before the program is even run |
| Integrated debugger | Lets you pause execution at any line, inspect variable values, and step through code one line at a time |
| Refactoring tools | Renames a variable or restructures code consistently across the entire project with one command |
| Built-in terminal | Includes a command-line panel so you never need to leave the IDE to run commands |
| Project management | Organizes large collections of files and makes navigating a large codebase much easier |
| IDE | Common Uses |
|---|---|
| Visual Studio Code | General-purpose; Python, JavaScript, web development |
| PyCharm | Python development |
| IntelliJ IDEA | Java and JVM languages |
| Eclipse | Java, C/C++ |
| Xcode | macOS and iOS development |
| Android Studio | Android app development |
For a small script or quick edit, a lightweight editor is fast and sufficient. For a large project with many files, an IDE's navigation and debugging tools save significant time. Many developers use Visual Studio Code for most of their work because it falls between a plain editor and a full IDE — lightweight enough to open quickly, but extensible with plugins to add IDE-like features when needed.
A terminal — also called a command line, command prompt, shell, or console — is a text-based interface for interacting with an operating system. Instead of pointing and clicking, you type commands. The shell reads your command, executes it, and prints the result as text. The $ symbol is the prompt — the shell's signal that it is waiting for input.
Many essential developer tools have no graphical interface. They are designed to be used from a terminal. This is not a limitation — it is a deliberate choice with real advantages.
| Advantage | Explanation |
|---|---|
| Scriptable | Commands can be combined into scripts that run many steps automatically, enabling automation and repeatable workflows |
| Remote-friendly | You can connect to a server anywhere in the world and run commands through a terminal connection, even without any graphical interface |
| Consistent | A text command behaves the same way regardless of screen size, operating system theme, or window arrangement |
| Precise | You specify exactly what you want in explicit text rather than navigating menus and clicking through dialog boxes |
| Command | What It Does |
|---|---|
ls / dir | List files and folders in the current directory |
cd foldername | Change the current directory to the named folder |
pwd | Print the path of the current working directory |
mkdir foldername | Create a new directory with the given name |
rm filename | Delete a file (permanently — no recycle bin) |
cp source dest | Copy a file from one location to another |
mv source dest | Move or rename a file |
cat filename | Print the contents of a file to the screen |
python script.py | Run a Python script file |
git status | Show the current state of a version-controlled project |
Unlike moving a file to the recycle bin in a graphical interface, the rm command in a terminal permanently deletes a file immediately. There is no undo and no recycle bin. Beginners should be careful with this command, especially with options like -r (which deletes a directory and everything inside it).
A package is a bundle of code — typically a library or tool — that has been packaged for easy downloading, installation, and use. Packages are the unit in which libraries are distributed. When a developer wants to use a library, they install it as a package using a package manager — a tool that handles downloading, installing, updating, and removing packages automatically.
After this, a Python program can use import requests to access the library's functions.
| Language / Environment | Package Manager |
|---|---|
| Python | pip |
| JavaScript (Node.js) | npm or yarn |
| Ruby | gem |
| Java | Maven or Gradle |
| Rust | cargo |
| macOS system tools | Homebrew |
A dependency is any library or package that your program relies on to function. If your program imports pandas, then pandas is a dependency. If pandas itself relies on NumPy internally, then NumPy is also a dependency of your project — an indirect (transitive) dependency. Most real software projects have dependency lists with dozens or even hundreds of packages, all of which must be installed for the software to work. Package managers handle this automatically.
Dependencies create efficiency — you get to build on other people's work — but they also create risk. A widely-used library with a security vulnerability makes every program that depends on it potentially vulnerable. Keeping dependencies updated is therefore a meaningful part of software maintenance, not a cosmetic concern. When a library is abandoned by its maintainers, programs depending on it may break as the world around them changes. In some cases, attackers have published malicious packages with names similar to popular libraries, hoping developers will accidentally install the wrong one — a type of attack called a supply chain attack.
When working on multiple Python projects, different projects may need different versions of the same library. A virtual environment is an isolated Python installation created for a single project, so each project can have exactly the dependencies it needs without conflicting with other projects on the same computer.
Virtual environments are a practical example of the broader computing idea of sandboxing — keeping different processes or environments separate so they cannot interfere with each other. The same principle applies in operating systems (processes cannot access each other's memory), in web browsers (one tab cannot read another's data), and in cloud computing (one customer's workload is isolated from another's).
analyze.py.Imagine working on a programming project over several weeks, making changes every day. Then you discover that something you changed two weeks ago broke a feature that was working fine before. Can you get the old version back? Without version control, the answer is probably no — unless you manually saved backup copies at every step. Even then, finding the right backup and understanding exactly what changed is difficult and unreliable.
Version control is a system that tracks changes to files over time. It records what changed, when it changed, and who changed it. It allows developers to view the complete history of a project, revert files or the entire project to any earlier state, see exactly what was different between any two points in time, work on new features without risking the stable version, and merge changes from multiple people working simultaneously.
Git is by far the most widely used version control system today. It was created in 2005 and is now used by virtually every professional software team in the world. Git tracks changes in a repository (often called a "repo") — a directory where Git watches every file for changes.
Commit — A saved snapshot of the project at a particular moment, with a message describing what changed. Every commit has a unique identifier, a timestamp, the author's name, and a message.
Branch — A separate line of development. You can create a branch to build a new feature without affecting the main working version, then merge the branch back when the feature is ready.
Merge — The act of combining changes from one branch back into another, integrating completed work into the main codebase.
commit 4a3f8e2
Author: Priya Sharma <[email protected]>
Date: Mon Oct 14 14:32:17 2024
Add input validation to login form
Users now see a clear error message if they enter
an email address without an @ symbol.
4a3f8e2), the author, the timestamp, and the message describing what changed and why.Branches allow parallel development. While one developer is building a new feature on a branch, the main branch stays stable. When the feature is complete and tested, the branch is merged back in. If the feature is abandoned, the branch is simply discarded without the main branch ever being affected. Most professional teams have conventions for branching: what the main branch represents, how long feature branches should live, and who reviews changes before they are merged.
Even individual developers working alone benefit enormously from version control. The complete recoverable history of a project, the ability to experiment on branches without risk, and the habit of writing commit messages that explain decisions are all valuable regardless of team size. Developing good version control habits from the start is one of the most impactful things a new developer can do.
Open-source software is software whose source code is publicly available. Anyone can read it, study it, use it, and — under certain license terms — modify and distribute it. This contrasts with proprietary software, where the source code is kept private by its owner. When you install proprietary software, you receive only the compiled, executable form; you cannot read or modify the underlying instructions.
Open-source software is remarkably central to modern computing. The Linux operating system — which powers most of the world's servers, Android phones, and supercomputers — is open source. The Python programming language and its standard library are open source. Most popular web servers, databases, machine learning frameworks, and development tools are open source. The infrastructure of the internet itself is largely built on open-source software.
| Benefit | Explanation |
|---|---|
| Transparency | Anyone can inspect the code to verify it does what it claims, check for hidden behavior, and identify security vulnerabilities |
| Collaboration | Developers worldwide can contribute fixes, improvements, and features — creating a collective quality that no single company can match |
| Cost | Open-source software is typically free to use, lowering barriers for students, researchers, small businesses, and developers everywhere |
| Education | Reading real, production-quality code is one of the best ways to learn programming — open source makes this possible |
| Longevity | Proprietary software disappears when its company stops supporting it; open-source software can continue to be maintained by its community indefinitely |
Not all open-source licenses are the same. A license specifies the rules for using, modifying, and distributing the software. Understanding licenses matters when building software professionally — using code without reading its license can create legal problems.
| License | Key Rules |
|---|---|
| MIT License | Very permissive. Use, modify, and distribute for almost any purpose — including in proprietary products — as long as you include the license notice. |
| GNU GPL (General Public License) | "Copyleft" license. If you distribute software that includes GPL code, you must also make your source code available under the GPL. Improvements must stay open. |
| Apache License 2.0 | Similar to MIT but includes explicit patent grants, providing additional legal protections. |
Contributing to open-source projects is both a learning opportunity and a way to build a portfolio of real work. Contributing can mean reporting bugs, improving documentation, fixing known issues, or adding new features that the community has requested. For developers early in their careers, an open-source contribution provides concrete evidence of skills that no class grade can fully convey.
Software reaches users in several forms. App stores (on phones and increasingly on desktops) distribute software through curated catalogs that handle downloading, installation, and updates — and provide some degree of security review. Package managers like pip, Homebrew, or apt install software from the command line and are the standard approach for developer tools. Traditional installer files (such as .exe or .dmg) walk the user through an installation process that copies files to the right locations and configures the software. For open-source software, some developers download and compile the source code themselves — less common for end users, but normal in certain developer workflows.
Installing software is only the beginning. Maintenance is ongoing and necessary. Updates fix bugs, patch security vulnerabilities, and add features. Keeping software updated is one of the single most impactful things users and administrators can do for security. Dependency updates matter even when a program's own code has not changed, because vulnerabilities in dependencies affect every program that uses them. Monitoring means watching a running application for errors, slow performance, or unexpected behavior. When something goes wrong, logs — timestamped records of events — are typically the primary tool for diagnosing the problem.
Professional software teams typically run their software in multiple environments, each serving a distinct purpose.
| Environment | Purpose | Who Uses It |
|---|---|---|
| Development | Where new code is written and first tested; it is acceptable — expected — for things to break here | Individual developers |
| Testing / Staging | A separate system that closely mirrors the real deployment; new versions are tested here before reaching users | QA teams, developers |
| Production | The live system that real users interact with; changes here must be careful and controlled | End users |
This separation is a practical application of risk management: problems discovered in development or testing do not affect real users. Skipping this separation — pushing changes directly to production without testing — is a well-known source of major software failures.
A common and costly mistake is making changes directly in the production environment without first testing in a staging environment. Even small changes can have unexpected effects. The staging environment exists precisely to catch these problems before they affect real users. Professional teams treat direct production changes without testing as a serious process failure.
fix-login-bug, makes three commits on it, and then merges it back into main. Draw a simple diagram of this process showing the branch diverging and then rejoining the main line.requirements.txt file in a Python project. Why is it important that it specifies exact version numbers rather than just package names?Study this diagram and answer the questions that follow.
requests? How does it arrive in a developer's project?requests.get(...) do? Which part of the software ecosystem does this interact with?.json() do to the response object?"temp" to "temperature", what would happen when this code runs? What type of error is this?requirements.txt file used for?==2.31.0) rather than just listing the package name?requests version 2.31.0 has a security vulnerability. What should you do? Why does this vulnerability affect your program even though you did not write any requests code yourself?pandas==1.5.0 and the other needs pandas==2.1.0. What tool prevents these from conflicting, and how does it work?Answer these questions in your own words without referring to the chapter.
pandas package using pip?fix-login-bug, makes three commits on it fixing a bug, and then merges it back into main. Draw a simple diagram of this process.git log and see three commits: "Add user profile page" (2 days ago), "Fix broken image links" (4 days ago), and "Initial commit" (1 week ago). You discover that the "Add user profile page" commit introduced a bug. Describe what version control allows you to do about this. What information in the log helps you?ModuleNotFoundError: No module named 'requests'. What is most likely wrong, and how can it be fixed?Each situation below contains a mistake or misunderstanding. Identify the problem and explain what should have been done instead.
A student installs a library using pip and then tries to import it in their script. The script fails with ModuleNotFoundError. They are confused because they definitely ran pip install. Later they discover they have two versions of Python on their computer, and pip installed the library for a different Python than the one running the script.
What went wrong? How do virtual environments prevent this kind of problem?
A developer is adding a new feature and makes all their changes directly on the main branch instead of creating a feature branch. Halfway through, they realize the approach is not working and they want to go back to the version before they started. They have made 12 commits since then, mixed with earlier stable work.
How would this situation be different if they had used a branch? What options do they have now?
A developer hardcodes a secret API key directly into their Python source code and then pushes the code to a public GitHub repository. Within hours, automated bots have scanned the repository, found the key, and used it to make thousands of dollars of API calls on the developer's account.
What serious mistake occurred? What should the developer have done to keep the key secret?
A developer hears about a new Python version with useful features. Eager to use them, they upgrade Python directly on the production server that thousands of users depend on. After the upgrade, the application crashes. Error logs show that a key dependency is not compatible with the new Python version.
What went wrong in the process? Describe the correct process they should have followed using the three environments (development, staging, production).
requirements.txt file for a Python project that reads a CSV file, calculates statistics, draws a bar chart, and sends the results to a web API. You do not need real library names — describe what kind of library you need for each task, then pick a plausible name for each.These questions connect Chapter 17 to earlier chapters.
python script.py, which component of the operating system is responsible for loading the Python interpreter into memory and starting it as a process?status equals "active". Express this filtering operation in plain English as a data processing step, using the vocabulary from Chapter 14's discussion of filtering and sorting data.Chapter 17 explained the software ecosystem: what software is, how it is built, and the tools developers use every day. Chapter 18 introduces a specialized but essential category of software: databases and information systems — the systems designed specifically to store, organize, retrieve, and protect large amounts of structured data. You have already encountered files and tables in Chapter 14. Chapter 18 goes deeper: it explains how relational databases organize data into tables with defined relationships, how queries allow powerful and flexible retrieval, how SQL (Structured Query Language) expresses those queries in a standardized form, and how database management systems ensure that data remains consistent, backed up, and reliable even under heavy use. You will also see how information systems — combinations of databases, applications, people, and processes — power the organizations that shape daily life: hospitals, banks, libraries, schools, governments, and the apps that connect them.