Here’s what you should be fairly comfortable with, before starting this tutorial:

Setup

First thing is to download the getting started files. I am going to pull down my own customized create-react-app setup from GitHub. Here are the terminal commands I use to get the hello world webpage:

git clone https://github.com/sojohnnysaid/custom-create-react-app.git hello-world
cd hello-world
yarn install
yarn develop

You should now see a browser window with some text. In your IDE you can use the hello-world folder as your root directory. Everything we code from this point on is in the src/index.js file.

What is JSX? First example without JSX

JSX stands for javascript XML and is a shorthand for the way React creates custom HTML elements. For example, In React, without JSX:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'


ReactDOM.render(React.createElement('App',{},'Hello! No JSX being used'), document.getElementById('root'))

This will render the text to the browser, and if we inspect we can see it is wrapped in tags. We use uppercase in our javascript for elements to differentiate them from normal HTML elements.

Adding a class based component

So far we’ve added an element directly into the ReactDOM render method. Let’s create a class with it’s own render() method and insert that into the reactDOM method as a variable.

Instead of ‘App’ as a string we will pass it directly as a class:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
    	return React.createElement('div', null, 'component class no JSX')
    }                                   
}

ReactDOM.render(React.createElement(App, null, null), document.getElementById('root'))

This puts our string inside of div tags. We can also change it to h1 tags or any kind we want:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        //wrapping our string in h1 tags instead
    	return React.createElement('h1', null, 'component class no JSX')
    }                                   
}

ReactDOM.render(React.createElement(App, null, null), document.getElementById('root'))

The createElement method takes 3 arguments – component, props, and children. This method is flexible and allows you to pass a custom element name for the component or an entire class using the name like we’ve shown above. Props are properties and they allow us to manipulate data from outside of our class. Finally children as basically what is inside our component. Our component is like the parent. If we look at the example above our h1 tag is the parent and inside is the children, our string ‘component class no JSX.’

More about props

Not a huge fan of shortening things because it adds confusion. Props is short for properties, and it allows us to fill in data. What are we filling? A class is just a blueprint for an object instance. I’m going to change our tag to a custom tag called ‘App’ instead of the standard h1 tag. In our class we can create placeholders when we add our string inside the parent like this:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
    	return React.createElement('App', null, `component class with a ${this.props.text} but no JSX`)
    }                                   
}

ReactDOM.render(React.createElement(App, {text: 'property'}, null), document.getElementById('root'))

If you inspect the webpage, notice how was now have custom tags. Our class blueprint now has a property called ‘text’ which we can access using the props argument in React’s createElement method. The second argument to the createElement is always an object and we include the property name and the value we want to pass in when our new object is created from our App class. We don’t have to pass a third argument when we use a class in the first argument, it will pass null by default if we left it out. This works the same:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
    	return React.createElement('App', null, `component class with a ${this.props.text} but no JSX`)
    }                                   
}
//removed null and completely left out the children argument
ReactDOM.render(React.createElement(App, {text: 'property'}), document.getElementById('root'))

One last thing before we add our JSX: What is Babel?

Babel does 2 things, first it converts our code from the latest javascript syntax, to a version of javascript every major browser can render. So currently ES6 code gets transpiled to ES5 syntax. The second thing Babel does is convert JSX into javascript. It does this using rules setup by React in a special configuration that Babel reads in. This plugin is working behind the scenes so when we write any JSX it turns the code into similar code I’m writing in the examples above using the createElement method. So why do we even want to write JSX at all?

Finally let’s use JSX

I’m going to remove the text property for a moment and just pass a simple string again. Let’s take our App classes render method and return JSX instead of using React’s createElement method:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
		return <h1>Hello JSX!</h1>
	}                                   
}

ReactDOM.render(React.createElement(App, null), document.getElementById('root'))

On line 7 you can see we can pass in JSX without quotation marks, because JSX is not a string, it is a mix of JS and XML-like markup. Behind the scene the createElement is behind called, after some time learning the flow it will become clear JSX is easier to read and maintain. Notice the last line we are still using createElement for App, so let’s change that but just passing in a custom tag like this, because JSX allows us to write a single self closing tag to do the same thing:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
		return <h1>Hello JSX!</h1>
	}                                   
}

ReactDOM.render(<App />, document.getElementById('root'))

Passing props using JSX

Now that we are using JSX, how do we pass a property from reactDOM.render to the App class so it can create an object that will become our component on the webpage.

We’ll pass in what looks like an html attribute like class or onclick, but this will be a custom attribute that matches the property in our App class “blueprint.” Here we’ll use text again as our property name:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
		return <h1>Hello JSX! {this.props.text}</h1>
	}                                   
}
//removed null and completely left out the children argument
ReactDOM.render(<App text='Hello props!'/>, document.getElementById('root'))

Anything we add in between JSX that is javascript like this.props.text has to be wrapped in curly braces. This is what tells Babel to stop interpreting JSX and run javascript code.

This does not mean we have to do this outside of the JSX. You can write normal Javascript everywhere else like this:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        const text = this.props.text
		return <h1>Hello JSX! {text}</h1>
	}                                   
}

ReactDOM.render(<App text='Hello props!'/>, document.getElementById('root'))

I’ve created a text variable and am passing it inside the JSX. I use curly braces, which will interpret the variable based on what I pass to my JSX:

Working with multiple properties and arrays

Let’s add a second property called seasons and we’ll pass an array of the 4 seasons:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        console.log(this.props.seasons)
        const text = this.props.text
		return <h1>Hello JSX! {text}</h1>
	}                                   
}

const seasons = ['winter','spring','summer','fall']
ReactDOM.render(<App text='Hello props!' seasons={seasons}/>, document.getElementById('root'))

notice the curly brackets around the seasons value inside of the JSX . Just remember anytime you introduce a javascript expression inside of JSX it has to be wrapped in {}brackets

So we’ve passed this to our class and it is now available in our App class this.props object. Let’s console.log and inspect the browser when the page loads:

So let’s display them on the page using a standard unordered list using these HTML elements:

Quick note! When you return something that requires multiple line breaks, remember to wrap your code in parenthesis:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        const seasons = this.props.seasons
        const text = this.props.text
		return (
			<ul>
				<li>{seasons[0]}</li>
				<li>{seasons[1]}</li>
				<li>{seasons[2]}</li>
				<li>{seasons[3]}</li>
			</ul>
		)
	}                                   
}

const seasons = ['winter','spring','summer','fall']
ReactDOM.render(<App text="Hello text property!" seasons={seasons}/>, document.getElementById('root'))

remember even though it looks like normal HTML it is in fact JSX so we need those brackets for any javascript code inside the JSX.

What if we pass the whole array like this: {seasons}

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        const seasons = this.props.seasons
        const text = this.props.text
		return (
			<ul>
				<li>{seasons[0]}</li>
				<li>{seasons[1]}</li>
				<li>{seasons[2]}</li>
				<li>{seasons[3]}</li>
                <li>{seasons}</li>
			</ul>
		)
	}                                   
}

const seasons = ['winter','spring','summer','fall']
ReactDOM.render(<App text="Hello text property!" seasons={seasons}/>, document.getElementById('root'))

Running this gives us our list and the list item is each season bunched together with no spaces. So JSX will even evaluate an array into a string.

Now we can leverage this by creating a new array with our season names, but with the tags wrapped around each one.

Since we’re doing the exact same thing for each item in the array to create a new one we can use the map method which is a built in array method.

Remember we don’t need brackets outside of JSX only in between the tags. First let’s create our new array using map:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        const seasons = this.props.seasons
        const seasonsList = seasons.map((name, index) => <li>{name}</li>)
        const text = this.props.text
		return <ul>{seasonsList}</ul>
	}                                   
}

const seasons = ['winter','spring','summer','fall']
ReactDOM.render(<App text="Hello seasons!" seasons={seasons}/>, document.getElementById('root'))

Let’s add back in our text property at the top to give our list a title:

// index.js
import React from 'react'
import ReactDOM from 'react-dom'

class App extends React.Component {
	render() {
        const seasons = this.props.seasons
        const seasonsList = seasons.map((name, index) => <li>{name}</li>)
        const text = this.props.text
		return (
        	<div>
            	<h1>{text}</h1>
            	<ul>{seasonsList}</ul>
            </div>
        )
	}                                   
}

const seasons = ['winter','spring','summer','fall']
ReactDOM.render(<App text="Hello seasons!" seasons={seasons}/>, document.getElementById('root'))

Notice how I wrap everything in a signal parent tag? Whatever we return from our class has to be wrapped in one single tag. If we try to run this without the we get an error for trying to return two parent tags. Always one parent and as many children as we want inside.

That is our entire program! Hope all of that made sense. This is just the start but the more comfortable you feel now the better as things become more spread out and complicated. React is not a framework, it is a library that gives you access to methods. Also React has a plugin which works behind the scenes with Babel to change our code on the fly when we save. Webpack is what is serving our page to the browser when we are developing.

What is Webpack?

Webpack is a tool that takes your files and packages them into a folder that you can send to your server. This is the production ready version of your application is commonly known as a “build.” It also runs the development server when you run ‘npm start’ or ‘yarn start.’

I use yarn so i’m going to create a ‘build’ folder that is the production ready version of my web application. This folder can be sent to a server which users can request to use your web app. Everything is self contained and create to be as efficient and fast as possible.

I’ll run ‘yarn build’ and in the terminal it will take about a minute to create my production build…

to run my build I will need to install the yarn serve package by typing:

‘yarn global add serve’

I will then cd into the build directory

and finally run the command ‘serve’

now my webapp should appear in the browser again. You might notice also it is a little faster to load up. This code is not to be touched. If you need to make a change go back to our index.js and re-build again to have a production ready app!

Conclusion

React is confusing at first. I’ve found a lot of tutorials really try to cram too much into an introduction, when the code above is all you need to start learning. Learning and getting comfortable is the first step, trying to build an app and learn React at the same time is only going to lead to headaches 🤦‍♂️

Once you feel really comfortable with what we’ve written so far, check out the next tutorial. We’re going to build a simple input that updates some text in realtime, then a simple button to clear our text. This is the reactive part of react!

I made an initial attempt that went very poorly. I read the CS50 assignment points and went through building the app page for page. Make all the code to register, then make all the code to get a quote, then buy, then sell. I started getting very picky about how the user got feedback when something was posted. I wanted nice banners to show things were successful or unsuccessful. I wanted javascript on the front end to update values. Everything became a big mess.

So i deleted it all. And pulled out some post-it notes, and a pencil.

I made 3 stacks:

• Frontend
• Database
• Logic

Then I started writing down all the tasks as I read them on the CS50 assignment page. Each post was very simple and specific. On the top left corner of each post-it I put the first letter of what stack it would go on. On the top right I put the first letter for the webpage (register, buy, sell, quote etc).

A post-it might be “create view template register.html,” which would be a F on the top left.

another might be “create model for users,” which would be a D.

“create get_balance method for users,” another D.

What I found was the Frontend stack could be done very quickly because I noticed there were similar tasks. They weren’t directly connected the way a user thinks when reading about the app. They were literally the same task just for a different page of the app.

So my advice would be to write out as many tasks as possible, sort them in this way, and try to develop frontend -> database -> logic.

For the database I highly recommend creating a new file called models.py and creating a class for each part of your app. I created users, transactions and portfolio. Here is what that looks like:

During the database phase I recommend using the python terminal REPL. If you go into the directory that holds application.py for the flask app, you can actually test out any function, class or method right in the terminal. Here is an example of me using the terminal to test some of the user database methods:

type python -i application.py. This is interactive mode and makes testing models easier so you don’t have to submit forms.

When you finish the database make sure to add some fake data to save time during the logic portion.

Logic is something like “if the user does not have enough cash to by stocks return apology.”

API changes

In the helpers file on line 42 I changed the url because the one given doesn’t work. Here is the actual api now https://iexcloud.io/docs/api/#quote

change to this on line 42:

        response = requests.get(f"https://cloud.iexapis.com/stable/stock/{urllib.parse.quote_plus(symbol)}/quote?token=YOUR_FREE_APITOKEN")

change your_free_api_token with your own token when you sign up.

Other quick notes

datetime: https://www.techonthenet.com/sqlite/functions/datetime.php

foreign keys: https://sqlite.org/foreignkeys.html

inner joins: https://www.sqlitetutorial.net/sqlite-inner-join/

Using the with keyword for opening and closing sqlite3

In python there is a way to open a sql connection and automatically close it using with.

Here would be the standard way of connecting to a database and closing it at the end.

import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()

c.execute('''CREATE TABLE stocks
             (date text, trans text, symbol text, qty real, price real)''')
c.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")

# Save (commit) the changes
conn.commit()

# close the connection if we are done with it.
conn.close()

Now using with:

import sqlite3
  with sqlite3.connect('example.db') as conn:
    c = conn.cursor()

    c.execute('''CREATE TABLE stocks
                 (date text, trans text, symbol text, qty real, price real)''')
    c.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")

    # Save (commit) the changes
    conn.commit()

    # connection automatically closed once we leave this indented section of code

Questions?

This was a much bigger project, so if you have any questions please let me know and i’ll do my best to answer. My code will be up on github:

Cookies and sessions start off this weeks lecture notes. The important thing to remember is http protocol is a stateless protocol. This means traditionally you request a webpage, the server gives you the page data and nothing else happens. Similar to going to a store, buying an item and leaving. The cashier doesn’t remember you next time. That is what stateless means.

Cookies and sessions allow the experience to be more stateful, meaning when you request a webpage, a cookie is inside that request header giving the server some unique info about you. This returns a webpage made only for you. Back to the store analogy, if every Monday you bought eggs, and the store had a way of knowing it was you right when you walked in, someone would just bring you eggs or even have them ready for you in advance at the register.

Here is the store demo code for application.py

On line 8 setting the SESSION_PERMANENT value to False means we do not store the session after the browser has been closed. If you quit the browser program and re-open it, the cart values will not be saved. However if the browser is still open and you close just the window, or tab, the cart values will persist. The SESSION_TYPE we are using is filesystem, which means our session values are stored on the server (in a folder called flask_session).

MVC – Model View Controller

MVC is a way of organizing pieces of a web application. The model is where all the database calls are made (variables being used in SQL statements most of the time). The View is for templating, like jinja2, the look of your pages. The controller is usually a main routes page like application.py, where decisions are made based on what the user is doing.

A Simple Example

I downloaded the lecture.db file from the notes and put it in the same directory as this script:

from cs50 import SQL

db = SQL("sqlite:///lecture.db")

# Query database for all albums
rows = db.execute("SELECT * FROM Album")

# For each album in database
for row in rows:
	# Print title of album
	print(row["Title"])

In my terminal I got this result:

Using sys.argv[1] we can pass variables into the command line which get substituted.

like this:

rows = db.execute("SELECT * FROM Album WHERE Title = :t", t=sys.argv[1])

Its important to remember we don’t use the normal formatting syntax f”my string {variable}” in this case. Because the variables are sanitized, meaning the removal of potentially harmful characters. SQL injection is a thing attacks can use to get into our databases, so it’s important to just know we need this sanitization before running our sql statements.

Adding Flask to the mix

So with a flask application we will use the lecture.db file to show album titles on a webpage:

from flask import Flask, render_template, request

from cs50 import SQL

app = Flask(__name__)

db = SQL("sqlite:///lecture.db")

@app.route("/")
def index():
    rows = db.execute("SELECT * FROM Album")
    return render_template("index.html", albums=rows)

running this (I had to remove eprint) will show a block of text on the browser, which is all the album titles. But the cool thing is we’re using a database. So we have state, which means we can save things for later =).

Query arguments

We can also pass query arguments from the URL using a GET request type:

@app.route("/")
def index():
    query = request.args.get("q")
    rows = db.execute("SELECT * FROM Album WHERE Title LIKE :q", q=query+"%")
    return render_template("index.html", albums=rows)

Above is a modified version of the example code. Using LIKE and a % symbol which is a wildcard, you can just use a single letter and it will return whatever starts with it.

So this URL: http://127.0.0.1:5000/?q=f

returns this:

So this is apparently enough for us poor online learning only souls to create an app for buying and selling stocks!

See you in the next one.

print(f"Hello {input('name: ')}")

Let’s quickly run through the “tougher” Déjà vu walkthroughs ( and then go deeper into the final similarities web app. Let’s start with the Mario challenge:

mario.py

So let’s build a pyramid shape in the terminal:

height = int(input("enter a number for your mario pyramid: "))
for i in range(2, height+2):
    print(('#'*i).rjust(height+1), end=" ")
    print('#' * i)

4 lines.

Google range() and it’s arguments, then google rjust() and it’s arguments. Use print statements to figure out what I did. If you have any questions try to say what this is doing out loud with a small number (like 2 for the user input). Comment below if you have any questions.

cash.py

cash = float(input("please enter cash amount to break down into coins (example 1.25): "))
coins = {"quarters": .25,"dimes":.10,"nickels":.05,"pennies":.01}
for key, coin in coins.items():
	(coins[key], cash) = (divmod(cash,coin)[0], round(divmod(cash,coin)[1],2))
print(coins)

Short but not impossible to read. Pull apart the packed variables and google divmod(), which is a very handy function in this case. Go back to my python article that goes through packing and unpacking if you need a refresher.

Caesar.py

import sys

if len(sys.argv) != 2: 
  print("Usage: python3 caesar.py key")
  sys.exit(0)
key = int(sys.argv[1])

message = input("enter a secret message: ")

def cipher(letter, key):
	if not letter.isalpha():
		return letter
	offset = 65 if letter.isupper() else 97
	return chr(((ord(letter) - offset) + key) % 26 + offset)

print(f"userinput: {message}")
print("decoded_text: " + ''.join([cipher(letter, key * -1) for letter in message]))
print("coded_text: " + ''.join([cipher(letter, key) for letter in message]))

Similarities (from scratch)

I’m going to do one from scratch then i’ll make it again using the CS50 instructions.

Here is the example image from the CS50 website:

Here was my result. You can check out the github repo as well. I styled the pages differently and added a number input for the substring page:

application.py:

from flask import Flask, render_template, redirect, request
from compare import compare_lines, compare_sentences, compare_substrings

app = Flask(__name__)

@app.route("/", methods=["GET","POST"])
def index():
    if request.method == "POST":
        file_one = request.files["file_one"]
        file_two = request.files["file_two"]
        return render_template("index.html", results=compare_lines(file_one,file_two))
    return render_template("index.html", results=None)



@app.route("/sentences", methods=["GET","POST"])
def sentences():
    if request.method == "POST":
        file_one = request.files["file_one"]
        file_two = request.files["file_two"]
        return render_template("sentences.html", results=compare_sentences(file_one,file_two))
    return render_template("sentences.html", results=None)



@app.route("/substrings", methods=["GET","POST"])
def substrings():
    if request.method == "POST":
        file_one = request.files["file_one"]
        file_two = request.files["file_two"]
        return render_template("substrings.html", results=compare_substrings(file_one,file_two, int(request.form.get('length'))))
    return render_template("substrings.html", results=None)


if __name__ == '__main__':
    app.run

The index route goes straight to the matching lines page. All forms send a POST request to the same URL, so i’m checking for the request type using the IF statement. At the top i’ve imported my own file compare, which has the functions I need to process the files that are being uploaded. Also notice to get the files I use request.files and then pass in the form’s name value. This will return what flask calls a filestorage object. This thin wrapper allows us to call read() which is still in a format that is not decoded because it doesn’t assume the file is text.

Here is the compare.py file so you can see the function I use to get the text from each file:

from nltk.tokenize import sent_tokenize, word_tokenize

def get_file_data(file_one, file_two):
    #takes filestorage type of bytes and turns file data it into a string
    data = [file_one.read().decode('utf-8'), file_two.read().decode('utf-8')]
    data = [file.replace('<', '&lt;').replace('>', '&gt;') for file in data]
    return data


def get_similarities(file_one, file_two):
    return set(file_one).intersection(file_two)


def format_html(file_one, file_two, similarities):
    #adds span tags and \n re-added if identical lines otherwise just the line with \n re-added
    similarities = list(similarities)
    files = []
    for item in similarities:
        file_one = file_one.replace(item, f"<span>{item}</span>", 1)
        file_two = file_two.replace(item, f"<span>{item}</span>", 1)
    files.append(file_one)
    files.append(file_two)
    return files

def set_files_as_list_of_strings(formatted_files):
    return [''.join(formatted_files[0]), ''.join(formatted_files[1])]


def compare_lines(file_one, file_two):
    data = get_file_data(file_one, file_two)

    #split string into list of strings wherever a linebreak is found
    file_one = data[0].split('\n')
    file_two = data[1].split('\n')

    #set intersection returns a set of all identical lines between the lists
    similarities = get_similarities(file_one, file_two)

    #adds span tags and \n re-added if identical lines otherwise just the line with \n re-added
    formatted_files = format_html(data[0], data[1], similarities)

    #return as a list of 2 strings
    return set_files_as_list_of_strings(formatted_files)




def compare_sentences(file_one, file_two):
    data = get_file_data(file_one, file_two)

    #TO DO splits data up into sentences
    file_one = sent_tokenize(data[0])
    file_two = sent_tokenize(data[1])

    #set intersection returns a set of all identical sentences between the lists
    similarities = get_similarities(file_one, file_two)

    #adds span tags and \n re-added if identical sentence otherwise just the sentence with \n re-added
    formatted_files = format_html(data[0], data[1], similarities)

    #return as a list of 2 strings
    return set_files_as_list_of_strings(formatted_files)




def compare_substrings(file_one, file_two, length):
    data = get_file_data(file_one, file_two)

    #split string into list of substrings based on user input
    file_one = [char for item in word_tokenize(data[0]) for char in item]
    file_two = [char for item in word_tokenize(data[1]) for char in item]

    slices_one = []
    for i in range(0,len(file_one), length):
        if len(file_one[i:i+length]) == length and ''.join(file_one[i:i+length]).isalpha():
            slices_one.append(''.join(file_one[i:i+length]))

    slices_two = []
    for i in range(0,len(file_two), length):
        if len(file_two[i:i+length]) == length and ''.join(file_one[i:i+length]).isalpha():
            slices_two.append(''.join(file_two[i:i+length]))


    #set intersection returns a set of all identical substrings between the lists
    similarities = get_similarities(slices_one, slices_two)

    print(similarities)

    list_of_substrings = ['\n'.join(similarities), '\n'.join(similarities)]

    #return as a list of 2 strings
    return set_files_as_list_of_strings(list_of_substrings)

On line 5 you can see where I call read() then decode(‘utf-8’) to get the files actual text.

When the user posts the data we’re not able to immediately get to the text inside the submitted files. My goal is for our data variable to just hold the file text as a string.

Using read() with no argument, I save all the form data to memory until EOF. This goes back to the days we were messing with file handlers in C. That returns a byte object. In python a byte object represents sequences of integers which are usually ascii characters. We can use the decode method to get back a string. I’ve passed ‘utf-8’ as an argument, however it actually decodes to this by default too, so decode() would work just as well.

Once I get the files as strings I can start working on formatting the data. If you look at where it ends up in html you can see i’ve wrapped the text in <pre></pre> tags. Let’s look at templates/base.html on line 110:

<!DOCTYPE html>
<html>
<head>
    {% block head %}
	<title>Similarities{% block title %}{% endblock %}</title>
	<style>
	    {% block background %}
	    body{
            background: #606c88;  /* fallback for old browsers */
            background: -webkit-linear-gradient(to right, #3f4c6b, #606c88);  /* Chrome 10-25, Safari 5.1-6 */
            background: linear-gradient(to right, #3f4c6b, #606c88); /* W3C, IE 10+/ Edge, Firefox 16+, Chrome 26+, Opera 12+, Safari 7+ */
        }
        {% endblock %}
        h1{
            line-height: 43px;
        }
        table, form{
            border-radius: 7px;
            box-shadow: 0px 12px 20px 1px;
            background: white;
            border-collapse: collapse;
            margin-left: auto;
            margin-right: auto;
            margin-top: 47px;
            font-size: 19px;
            line-height: 27px;
            width: 89%;
        }

        form{
            padding: 2px 0px 21px 39px;
        }

        input{
            font-size: 16px;
            margin: -14px;
        }


        table, th, td {
            border: 1px solid #191e2a;
        }

        th, td {
            padding: 15px;
            text-align: left;
        }

        span{
            background-color: yellow;
        }

        li a{
            text-decoration: none;
            padding: 16px;
            color: white;
            font-size: 27px;
        }

        li{
            flex: 3;
            text-align: center;
        }

        ul{
            display: flex;
            list-style-type: none;
            margin-top: 30px;
        }

        menu{
            padding-left: 0px;
        }

        pre{
            white-space: pre-wrap;
        }
        td{
            min-width: 45vw;
        }
	</style>
    {% endblock %}
</head>
<body>
    <menu>
    <ul>
        <li><a href="/">Similar Lines</a></li>
        <li><a href="/sentences">Similar Sentences</a></li>
        <li><a href="/substrings">Similar Substrings</a></li>
    </ul>
</menu>
    {% block content %}
    <form action={{ request.path }} method="post" enctype = "multipart/form-data">
      <h1>Upload two files to highlight similarities{% block h1 %}{% endblock %}</h1>
      <input type="file" name="file_one" id="file_one"><br>
      <input type="file" name="file_two" id="file_two"><br>
      {% block numberInput %}{% endblock %}
      <input type="submit" value="Submit">
    </form>
    <table>
        <tr>
            <th>
                Code Group One
            </th>
            <th>
                Code Group Two
            </th>
        <tr>
            <td id="groupone">
<pre>
{% block results_one %}{% endblock %}
</pre>
            </td>
            <td id="grouptwo">
<pre>
{% block results_two %}{% endblock %}
</pre>
            </td>
        </tr>
    </table>
{% endblock %}
</body>
</html>

Using jinja2 templating I can create a block that holds results_one and on line 116 I have another <pre> </pre> with results_two inside. Now let’s look at index.html which is the first page of the website, where I compare lines:

{% extends "base.html" %}

    {% block title %} - Lines{% endblock %}
    {% block background %}
        body{
            background-image: linear-gradient(to right, #ab7272 0%, GRAY 280%);
        }
    {% endblock %}
{% block head %}
    {{ super() }}
{% endblock %}

    {% block h1 %} - matching lines{% endblock %}
    {% block results_one %}{{results[0]|safe}}{% endblock %}
    {% block results_two %}{{results[1]|safe}}{% endblock %}
{% block content %}
    {{ super() }}
{% endblock %}

This is why jinja2 is so powerful. My base.html is 123 lines of code. My index.html, sentences.html and substrings.html pages are less than 20! If I have to make a change on all the webpages, base.html will be the one place to make it.

Using {% extends “base.html” %} allows me to reuse all the code and slip in custom variables by defining their values on each individual page. Look at line 14:

{% block results_two %}{{results[1]|safe}}{% endblock %}

I pass in the results from my compare function in application.py. Let me make a diagram so it’s easier to understand the flow:

This is generally what is happening. Things are being passed to do some processing work, but always end up back in application.py so the flask framework can do the rest for us.

I’m still learning as I go, and the tough part of this is all the different languages working together…the html, css, server calls, url requests and python code. Each is a journey on it’s own and don’t be discouraged if this made little sense. CS50 doesn’t cover html, or css very much so if you’re up to it check out teamtreehouse for video courses you can take on those topics.

Finishing the actual challenges

Back to similaries less, which is a much simpler challenge. We don’t have to make the flask app from scratch. Just update the helpers.py and index.html files with functions and a form. Here is what I ended up doing:

My drop down has a nifty javascript function inside it that will hide or show the numbers field if substring is selected. Also when the user hits the back button it resets the form using the script at the bottom.

So remember, a lot of these features i’ve added is not because i’m some all powerful genius. I am just not afraid to ask myself “what do I want this to do?” followed by googling it and not being too proud to use code from online. In these cases, if it works, great! The script at the bottom is straight from a stack overflow answer. I also forgot how to inline javascript in html so a quick google and you can see onchange on line 14 that goes into my javascript for changing the style of an element (which I also googled because I forgot for the 100th time, which is OK)

There is too much to remember, and too much changing in the world of programming, be a hunter of code examples and understand fundamentals to get things done and not get stuck with half finished projects.

Moving on to week 8!

So with a brief overview of html, css and python it is time to put everything together.

If you’ve never programmed something for the web just remember:

THE INTERNET IS JUST A BUNCH OF COMPUTERS ALL TALKING TO EACH OTHER

There is no special “cloud” machine or “server.” These are all just words that describe COMPUTERS. It’s not much different from sending a hand written letter from one house to another house. The mailman is just your internet cable and some electricity.

We have to command our computer to send information (think hand written letter) using a request called GET. So if you go inside your browser like safari, edge, chrome, firefox, whatever, that bar at the top for web addresses is where you send GET requests. You are saying “go GET me the information from google.com.” When you google cats the bar changes to some funky address that is saying “GET me cats.” (https://www.google.com/search?q=cats&oq=cats&aqs=chrome.0.69i59j0l4j69i61.3015j0j8&sourceid=chrome&ie=UTF-8)

On our computer we might have a word document locally. If we open it we get the information on the screen so we can read it. It is easy in our case to write more stuff onto the document and save it. How do we do this on the web?

There are a few different commands besides GET but the next one you should know is called POST. If I have data somewhere on the internet, meaning not on my computer but on the web, I can add data to it by using a POST type request. It is still a web address but it has some data tied to it that will be saved on the other end. You can also get a response from a POST request. Usually it will be a message that your data was either saved, or something went wrong.

Web requests have headers. I’m on Firefox so if I make a GET request, I can inspect by right clicking and choosing inspect. This is similar on a lot of browsers. Here is how I found the headers:

So we can see in my case we’re using HTTP/2.0 the response code was 200 which means all is good and our content type is text/html; charset=UTF-8.

UTF stands for Unicode Transformation Format and after spending the past hour looking this up I think I get how it works. I’m probably not 100% but this is the gist. Please leave a comment if you have any additional info on the topic that would help others. Here is an image I put together to help visualize what is happening:

utf-8 is the encoding lookup table that turns a character code into binary for a computer to read or pass back out to Unicode. Unicode (a character set) takes the character code and maps it to a lookup table that has the actual character we display on the page using a typeface or font.

I hope that made sense. If it doesn’t please let me know in the comments.

HTML

HTML holds 2 types of info. Info we see and info we don’t see. Here is a sample from the CS50 lecture notes:

<!DOCTYPE html>

<html>
    <head>
        <title>hello, title</title>
    </head>
    <body>
        hello, body
    </body>
</html>

The browser program is going to interpret this starting at the top. The first line lets the browser know we are dealing with HTML5, the current standard. Next we have <html> which starts and ends all the info we’re going to have on our page.

<head> is where the browser gets info regarding the title of our page, scripts we want to run, and CSS.

<body> is where we hold the material that can be viewed on the screen by our user. Inside the body we can have website titles, images, text, links and anything else you’ve ever seen on a webpage.

Notice how everything has a closing tag and has a nested structure.

CSS

CSS to style our page. We can change the character font, color, add borders, background images and even animations. When you hover your mouse over a button, and the button on the page looks slightly different when you hover over it, that is CSS. Here is some example code:

You can put css in the <head> section of your html file, or reference a separate .css file. The example file above can be opened using your browser without any other files. Try hovering over the button and last list item.

In CSS the first letter stands for cascading. It’s important you know that in css the rule closest to the bottom will always been the final say.

Notice on line 21 of my example I set the font to 1000000000px! Then on the following lines I change the font. So if a css rule is targeting the exact same element in the html, the last rule will win. Remember I wrote exact. If one rule is li and the next rule is li:first-child THEY ARE NOT EXACTLY THE SAME. So the cascading rule of last is not true.

To learn more I recommend W3Schools CSS tutorials, they are excellent.

Dynamic content with python

This look complicated but lets try to break this down. I was stumped at first but after posting some questions on reddit I figured it out:

# Implements a web server

from http.server import BaseHTTPRequestHandler, HTTPServer


# HTTPRequestHandler class
class HTTPServer_RequestHandler(BaseHTTPRequestHandler):

    # GET
    def do_GET(self):

        # send response status code
        self.send_response(200)

        # send headers
        self.send_header("Content-type", "text/html")
        self.end_headers()

        # write message
        self.wfile.write(bytes("hello, world", "utf8"))


# configure server
port = 8080
server_address = ("0.0.0.0", port)
httpd = HTTPServer(server_address, HTTPServer_RequestHandler)

# run server
httpd.serve_forever()

Ok after going through the notes I understand what is happening. Seeing a class with a do_GET method and not finding it being called in the program was really confusing at first.

do_GET gets called when the user goes to the browser and loads that webpage (makes a GET request).

wfile was confusing at first too, but it’s a file handler used by the HTTPServer class. It requires a bytes object so python has a helpful function that takes in a string and an encoding (utf-8). So we have above the example of ‘hello world’ but we can also create our own file handler, insert it into the bytes function with the read() method and serve up our text file to the webpage.

Here is that version:

# Implements a web server

from http.server import BaseHTTPRequestHandler, HTTPServer


# HTTPRequestHandler class
class HTTPServer_RequestHandler(BaseHTTPRequestHandler):

    # GET
    def do_GET(self):

        # send response status code
        self.send_response(200)

        # send headers
        self.send_header("Content-type", "text/html")
        self.end_headers()

        # write message
        # file handler gives us a stream. bytes turns it into a bytes object
        # we can pass the file handler, a string of text
        # just remember to pass the encoding type utf8!
        f = open("test.txt")
        self.wfile.write(bytes(f.read(), 'utf8'))
        f.close()


# configure server
port = 8080
server_address = ("0.0.0.0", port)
httpd = HTTPServer(server_address, HTTPServer_RequestHandler)

# run server
httpd.serve_forever()

I created a file called test.txt in the same directory for this test. Next I’ll try an html file:

# Implements a web server

from http.server import BaseHTTPRequestHandler, HTTPServer


# HTTPRequestHandler class
class HTTPServer_RequestHandler(BaseHTTPRequestHandler):

    # GET
    def do_GET(self):

        # send response status code
        self.send_response(200)

        # send headers
        self.send_header("Content-type", "text/html")
        self.end_headers()

        # write message
        # file handler gives us a stream. bytes turns it into a bytes object
        # we can pass the file handler, a string of text
        # just remember to pass the encoding type utf8!
        f = open("test.html")
        self.wfile.write(bytes(f.read(), 'utf8'))
        f.close()


# configure server
port = 8080
server_address = ("0.0.0.0", port)
httpd = HTTPServer(server_address, HTTPServer_RequestHandler)

# run server
httpd.serve_forever()

Here is my html file:

<!DOCTYPE html>
<html>
<head>
	<title>Wow CATS</title>
</head>
<body>
	<h1>CATS ARE AWESOME!!!</h1>
	<p>Hello world this is my cats page please subscribe for lots of cats</p>
	<br>
	<img src="https://inteng-storage.s3.amazonaws.com/img/iea/4N610VqxGJ/sizes/cat-cloning_resize_md.jpg">
</body>
</html>

The result:

So far we’ve created a static page but now we have to go even deeper and build dynamic pages. This is a pretty dense topic and we’ll soon see the differences. Here is a quick read that explains the differences well.

Flask – a python web framework

Make sure to type pip install flask in the terminal before getting started.

Let’s start with a simple example, straight from the Flask website, which you can find here.

I’ve made a folder called flask_example, and inside a file called mysite.py. Inside that file are a few lines of code:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello_world():
	return 'Hello, CS50!'

If I tried to run this in the terminal by typing flask run, I will most likely get this message:

 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
Usage: flask run [OPTIONS]

Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.

It wants us to create an environment variable, which is a variable our operating system holds (not our program). Flask reaches to the OS for this variable to reference our file.

We can create environment variables by using the export command in the terminal.

So lets create the FLASK_APP environment variable that is the name of our program. In this example the file i’m working in is called mysite.py

Now when I run the command flask run I get this:

If I hover of the website address and click open I get this:

Now that we’ve created this simple example lets follow the lecture notes.

Flask continued (lecture notes)

I’m going to rename mysite.py to application.py so it matches up with the notes. Then i’m going to create a folder called templates. The notes are a little confusing here but to clear it up here is the directory structure:

• flask_example
  • application.py
  • templates

So flask_example is the main folder, which has application.py and the templates folder inside of it.

first we have to make sure to reset our environment variable!!!!

In the terminal type export FLASK_APP=application.py

Next, inside the templates folder i’ll create a new file called index.html that has this html:

<!DOCTYPE html>

<html>
    <head>
        <title>hello, title</title>
    </head>
    <body>
        hello, body
    </body>
</html>

I’m going to replace the code in application.py with this:

from flask import Flask, render_template

app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/zuck")
def zuck():
    return render_template("zuck.html")

@app.route("/login")
def login():
    return render_template("login.html")

Next i’ll run the flask run command in the terminal:

Now inside of templates create a zuck.html file with this html:

<!DOCTYPE html>

<html>
    <head>
        <title>hello, title</title>
    </head>
    <body>
        hello, this is the zuck page!
    </body>
</html>

Enter flask run in the terminal again and open the website URL. Then add “/zuck” to the end of the url and you will get this result:

Error handling and tracebacks with flask

You can skip this and just run these commands:

touch ~/.bash_profile && echo export FLASK_DEBUG=1 > .bash_profile

Then restart the terminal for it to work. Click here for the next section of text.

This was a huge pain to get working with CS50 IDE. I spent about an hour on this because I really wanted that nice traceback feature right in the browser. Turns out there is another environment variable that needs to be set similar to FLASK_APP. I found this in the flask documentation:

To enable all development features (including debug mode) you can export the FLASK_ENV environment variable and set it to development before running the server:

$ export FLASK_ENV=development
$ flask run

(On Windows you need to use set instead of export.)

This does the following things:

1. it activates the debugger
2. it activates the automatic reloader
3. it enables the debug mode on the Flask application.

You can also control debug mode separately from the environment by exporting FLASK_DEBUG=1.

It turns out FLASK_ENV wasn’t the answer, I just needed that last little part FLASK_DEBUG=1. Or to be exact export FLASK_DEBUG=1

To make it more frustrating I had to do this every time I opened a new terminal. To fix this I wrote this in the terminal:

touch ~/.bash_profile && echo export FLASK_DEBUG=1 > .bash_profile

I’ve run into .bash_profile in the past and I recommend doing a quick google how it works when you get a chance.

Now open a fresh new terminal, close the old one, and try to go to the login.html (make sure in the templates folder you delete login.html so it is not there since we’re trying to mock an error situation). You should get that awesome jinja2 traceback!

Frosh IMs – what it means

I have no idea what Frosh is let me look that up…

Ok google tells me it is a slang word for a first year student (like a freshman).

Ok… fine.

IM’s stands for intramurals which are sports organized within an institution. Fun fact, the word is derived from the Latin intra muros which means “within walls.” Now that i’ve unpacked the heading of this section let’s get down to the programming part:

File structure is similar to the previous example:

• froshims0/
  • application.py
  • templates/
    • failure.html
    • index.html
    • layout.html
    • success.html

If we go through each file in the link provided here, we can go into each file, copy the code and run it to get the form:

Next I would like you to open the inspector in your browser by right clicking and choosing inspect. This is a little different for each browser. I’m using chrome in this case:

Next I want you to click on the “Network” tab:

A post request means the form values are sent in the HTTP body data. What does that mean? Even if we can’t see the body data doesn’t mean the other end doesn’t receive and read it. Lets think of it this way:

When I open my browser it’s like walking into a post office, where I get my mail. When I go to the counter, the clerk asks for information necessary to retrieve my mail. On my computer, when I type google.com it’s like asking the post office worker “please get me google.com.” Once that post office worker brings me an envelope let’s imagine it like this for google.com:

Now if I take this same idea and fill out a form on a webpage. I can use a GET request or a POST request. Using GET I pass in the form values right into the URL address. When filling out the frosh form using the method GET, and hitting submit the url will change to this:

https://792cdab9-edb1-4a25-a6df-87b8146cafcf-ide.cs50.xyz:8080/register?name=john+smith&dorm=Canaday

Notice the last part: ?name=john+smith&dorm=Apley+Court

So if we look back at the envelope image it might look like this:

This is usually ok for quick and dirty apps that don’t need to be secure with data. A form about if you like cats or not might be ok for the GET method.

Otherwise we have to use POST. So if i change my code back so I use the POST method, which is how it was written by default, my url looks like this after submitting:

https://792cdab9-edb1-4a25-a6df-87b8146cafcf-ide.cs50.xyz:8080/register

So where did my form values end up?

THE SAME PLACE WHERE YOUR BROWSER STORES PAGE DATA. Like LITERALLY THE SAME PLACE. I had no idea for the longest time, because you can’t see what the computer (server) sees on the other side. It uses that space because you can store tons more data. Think about how many images, videos, text you view on the webpage body. That allows forms to take advantage of all that space to then store things to the computer on the other end of the form submission.

So here is what a post request looks like:

Flask routes and more

Let’s look at the application.py file in our lecture notes:

from flask import Flask, render_template, request

app = Flask(__name__)

@app.route("/")
def index():
	return render_template("index.html")

@app.route("/register", methods=["POST"])
def register():
	if not request.form.get("name") or not request.form.get("dorm"):
		return render_template("failure.html")
	return render_template("success.html")

When we use from flask we’re saying from the flask file, give me access to the Flask class. Python classes by convention start with a capital letter. We are also importing the render_template function, and the request object, which has methods we can use to GET and POST things.

Next we have this funky @app.route(“/”), followed by a function def.

@app.route("/")
def index():
	return render_template("index.html")

This is called a decorator. Let’s break down what that means.

A quick peak at decorators, factories and wrapping functions.

Let’s jump right into this magic™

def returnPositiveNumber(number):
	if number > 0:
		return number
	else:
		return 1

Here we have a simple function. It will always return a positive number. If it’s zero or a negative number it will return 1. Easy. Our function takes in one arguement num as it’s only argument.

Python treats functions as first-class-objects, which means you can put them just about anywhere, including as arguments to other functions.

Check this next piece out:

def plusNineThousandDecorator(function):
	def wrapper(number):
		return function(number) + 9000
	return wrapper

PlusNineThousand takes a function as its argument. So we’re going to pass our returnPositiveNumber function into it. How does this look if we try to print something out?

SUPER IMPORTANT! When we use a function as an argument we are NOT calling the function, we reference it as an object which is super strange at first sight. NO () are used!

test = plusNineThousand(returnPositiveNumber)

Look back to our plusNineThousand definition and check out the last line. It returns decorator. So what is decorator? It’s a function. So test is a function. What does it do? How does it work? Take a look at it’s definition:

return function(number) + 9000

It takes the same argument as our original function. So we need to run it as test(num). But now it adds 9000 to the returned result.

Let’s try it out:

print(test(42))
#result is 9042

print(test(-1))
#result is 9001

test is now a function that takes our original function which returned num and added a piece of new functionality, which is adding 9000 to it and returning it back.

When using decorators it is common to modify the original function like this:

returnPositiveNumber = plusNineThousandDecorator(returnPositiveNumber)

Now our original function is what it used to be plus it adds 9000 to everything.

print(returnPositiveNumber(42))
#result is 9042

We can use some syntax sugar to make our ugly:

returnPositiveNumber = plusNineThousandDecorator(returnPositiveNumber)

Into this:

@plusNineThousandDecorator
def returnPositiveNumber(number):
	if number > 0:
		return number
	else:
		return 1

Make sure the decorator function is before the function you modify. Also the decorator @ has to be placed on the line before the function like shown above.

Here is the entire program:

Going deeper, we can assign arguments to our @plusNineThousandDecorator. We can only modify before and after our modified function so let’s try to add some html tags to the front and back of our number.

So we can not pass arbitrary arguments to our decorator. The decorator returns our original function wrapped in some other stuff.

Now we need a “factory” that returns a decorator while passing some arguments in the process.

So let’s “wrap” our decorator function in a decorator factory:

def decoratorFactory(htmlTag):
	def plusNineThousandDecorator(function):
		def wrapper(number):
			return f"<{htmlTag}>{function(number) + 9000}</{htmlTag}>"
		return wrapper
	return plusNineThousandDecorator


@decoratorFactory("h1")
def returnPositiveNumber(number):
	if number > 0:
		return number
	else:
		return 1


print(returnPositiveNumber(42))


# results
# <h1>9042</h1>

Now we can modify our factory argument, get back a decorator, which then wraps our function, and finally modifies it’s behavior.

This keeps our program dry, or without repetitive code that is difficult to keep track of. If one thing changes in our program we only want to go fix it in one and only one place. Here is a great article on this topic. Decorators help us achieve this goal when creating software.

Back to our flask app

So now that we know a little bit about decorators let’s look again at a basic flask application:

app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

The example above is from ains.co, and I highly recommend coding through both part 1 and 2 articles here.

What is happening behind the scenes is this (basically):

#credit goes to ains.co 
#please read the article on his site it's fantastic
#https://ains.co/blog/things-which-arent-magic-flask-part-1.html

class Flask():
    def __init__(self):
        self.routes = {}

    def route(self, route_str):
        def decorator(f):
            self.routes[route_str] = f
            return f

        return decorator

    def serve(self, path):
        view_function = self.routes.get(path)
        if view_function:
            return view_function()
        else:
            raise ValueError('Route "{}"" has not been registered'.format(path))


app = Flask()

@app.route("/")
def hello():
    return "Hello World!"

We haven’t used decorators in classes, but it’s basically the same except here we call a method inside of the Flask class.

The cool part is the self.routes{}.

Flask is storing each path (“/”) as a key, while the value is the decorator function itself.

It doesn’t go that final level deep to wrapper. In our example we used our modified function right away, but flask is saving this function to be used later in the serve method.

Once we make a request in our browser to the main domain (“/”), flask is going to check if that is in it’s self.routes dictionary. If it is it will return the function that is stored, which is hello().

If this is confusing do not worry. It is ok to just understand the results of the code and get a feel for how flask operates. This is just a look under the hood of the flask engine. Again here is the link if you want to learn (most of) the inner workings.

Getting form values from flask

Since our form action in the index.html <form> is <form action=”/register”… we have to let flask know about this url, otherwise known to flask as a route, and what we should do when the user submits our form to it. So we are sending our form message to that url using the POST method as shown below:

@app.route("/register", methods=["POST"])
def register():
    if not request.form.get("name") or not request.form.get("dorm"):
        return render_template("failure.html")
    return render_template("success.html")

Our function has some logic that guards against the user not submitting the correct values, and finally if the guard is passed we just return the success.html page found in our templates folder.

Some cool is the request object we have access to. This allows us to work with the form values that are specified in the index.html <form> element. These are

<input type=”text” name=”name”…

and

<select name=”dorm”>

using the name attribute we can use request.form.get(nameAttribute) which gives us the value we specify.

Now let’s take a look at the failure.html file to pull apart what is happening.

Flask templating

Take a look at the templates/failure.html file in the froshim0 flask application:

{% extends "layout.html" %}
{% block body %} 
You must provide your name and dorm!
{% endblock %}

The curly brace percent symbol syntax is a templating language called jinja2. This means we can write regular html, and when we need an expression like a variable we use the following syntax: {{ myVariable }}. When adding logic (for loops, conditionals etc), we wrap them in the curly brace percent symbol like this: {% logic here %}.

Here is what the failure.html page looks like:

{% extends "layout.html" %}
{% block body %} 
You must provide your name and dorm!
{% endblock %}

If we look into layout.html we only see this:

{% block body %}{% endblock %}

Because there is nothing between block body and endblock, layout.html is not actually adding anything to our failure.html file.

If we erase the jinja2 lines and leave only the plain text:

You must provide your name and dorm!

our webpage result is the same. Run the flask app in the terminal by typing flask run. Go to the webpage that has the form and leave it blank. Then click submit. We should get this.

Now let’s put the jinja2 lines back into failure.html.

Next go back to layout.html and add something like this:

{% block body %}
HELLO LAYOUT!
{% endblock %}

Now when we test the webpage again we get this:

So we don’t see HELLO LAYOUT!

This is because {% block body %}{% endblock%} is a placeholder for any file that extends layout.html. So if failure starts with the {% extends … %} expression that means anything within the {% block body %}{% endblock%} in the failure.html file will appear in the {% block body %}{% endblock%} inside the layout.html file.

Now let’s try adding some html to the outside of the expression in the layout.html file. Try this, save and re-run the flask app by canceling it in the terminal with ctrl-c, then re-running flask run.

Here is layout.html now:

HELLO START OF LAYOUT<br>
---------------------<br>
<br>
<br>
{% block body %} THIS WILL NOT RENDER {% endblock %}<br>
<br>
<br>
-------------------<br>
HELLO END OF LAYOUT

Now things are getting interesting. Every page in our app now has this text we’ve added:

Now we can put pieces of our webpage that repeat on many pages in one place, and extend them using the {% extends …%} method in jinja2.

Storing form data with flask

Flask allows us to access form fields when submitted by a user. Looking at application.py on line 1, we import request. Here is an updated version of our code:

from flask import Flask, render_template, request, redirect

app = Flask(__name__)

# Registrants
students = []

@app.route("/")
def index():
	return render_template("index.html")


@app.route("/registrants")
def registrants():
	return render_template("registrants.html", students=students)



@app.route("/register", methods=["POST"])
def register():
	name = request.form.get("name")
	dorm = request.form.get("dorm")
	
	if not name or not dorm:
		return render_template("failure.html")
	
	students.append(f"{name} from {dorm}")
	return redirect("/registrants")

Flask is instantiating the request object when the user goes to a webpage. So If I go to “/” or “/registrants” a new request object is made. This is because of threads and context. Threading means the program is doing several things at once, a good example is one user going to our “/” webpage and another user at a different computer going to “/register.” Each thread has different stuff going on. The context of the request is not the same. So flask hands us back a unique object with values that are relevant to the user’s request.

on line 6 we create a list of “students.” because this lives outside of all our routes and functions it will be saved for as long as we run our flask app. It’s stored in memory.

on lines 21 and 22 we create variables “name” and “dorm” which store the user input from our form. Our request object has a form object inside of it with a get method. This is kind of confusing because it’s not like get and post kind of get, it’s just the “get me the value” kind of get. To clarify:

• get POST request values
  • use request.form.get(field name=”thisvalue”)
• get GET request values
  • use request.args.get(field name=”thisvalue”)

Finally on lines 24 through 28 we guard against the user not inputing values in our form by checking if either name or dorm return false. If one or both return false in our if statement, we return our failure.html page using the render_template function.

If the user submitted all the form values we use our “students” list and use the append method to add the formatted string to our list. Finally we redirect the user to the “/registrants” page. This confused me because I didn’t realize the app route is going to “/register” and the redirect is to “/registrants.”

So normally we could pass a template using render_template. But I think it makes sense to use redirect to minimize code repetition. If the registrants template file ever had to change you would only have to change it in one place. The template should stay with the route!

Notice line 15 how we have students=students as an argument in the render_template fuction:

@app.route("/registrants")
def registrants():
	return render_template("registrants.html", students=students)

We’re creating a new variable for our render_template so it can access our students list. So the students on the right side is the one that has our name and dorms, while the one on the left is passed to our template. This is where jinja2 comes into play.

Go to templates/registrants.html

{% extends "layout.html" %}

{% block body %}

<ul>
    {% for student in students %}
    <li>{{ student }}</li>
    {% endfor %}
</ul>

{% endblock %}

On line 6 we are using a for student in students loop. The students is the list of student data we passed from our application.py!

Just like any python loop we express a single item in our list but creating a temporary variable, in this case we call it student (singular). Now within the loop we add it to our html list with double curly braces {{ student }}.

We have to end our for loop when using html and jinja2 so on line 8 we use the {% endfor %} statement.

Here is what happens after submitting the form two times:

Don’t forget using app.route doesn’t have to return a webpage! Notice how we don’t have a register.html file in our templates. This is because that route is only being used to process some values. Once we have them we can redirect them to an actual webpage. Each time the user goes to a different address it’s just running a function for us. That might be a webpage, other times it might just be doing some work behind the scenes.

Adding more features to the froshim form

Next up we add an email letting the user know they registered. Of course new features mean new import statements so let’s bring them in:

import os
import smtplib
from flask import Flask, render_template, request

We aren’t using the redirect method so we’ll leave it out this time.

Next we create our app object and the usual route to our root web address “/”:

import os
import smtplib
from flask import Flask, render_template, request

# Configure app
app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

Then we configure the “/register” route to send a confirmation email to the user:

import os
import smtplib
from flask import Flask, render_template, request

# Configure app
app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")


@app.route("/register", methods=["POST"])
def register():
    name = request.form.get("name")
    email = request.form.get("email")
    dorm = request.form.get("dorm")
    if not name or not email or not dorm:
        return render_template("failure.html")
    message = "You are registered!"
    server = smtplib.SMTP("smtp.gmail.com", 587)
    server.starttls()
    server.login("jharvard@cs50.net", os.getenv("PASSWORD"))
    server.sendmail("jharvard@cs50.net", email, message)
    return render_template("success.html")

Using the smtblib (SMTP library – simple mail transfer protocol) we can log into an email account, and send emails to users through it. Just like you would from your own email account.

Using the os library allows us to use operating system variables and methods from within our program. We don’t want a plain text password in our program, so we can have the os library call it from an environment variable we set outside of the program. I’m using a mac, so in the terminal I would type export PASSWORD=mypassword.

This script doesn’t work without a few changes. Can you try to use your email to send a success message back to the same email?

I had to add the email input in the index.html file, create an environment variable with my email password then update the script with my email address.

Finally google doesn’t allow apps like this to work by default. So do a quick search for allowing less secure apps in gmail to briefly allow the app to send the email to see it working.

Using the csv library

Here is another example where we save the registrants to a csv table file:

from flask import Flask, render_template, request
import csv

app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/register", methods=["POST"])
def register():
    if not request.form.get("name") or not request.form.get("dorm"):
        return render_template("failure.html")
    file = open("registrants.csv", "a")
    writer = csv.writer(file)
    writer.writerow((request.form.get("name"), request.form.get("dorm")))
    file.close()
    return render_template("success.html")

Getting csv (comma separated values) data can be really handy to users who have to filter data later in a spreadsheet. Python3 comes with a great library for this called csv.

On line 14 we use the open function to create our csv file, and the “a” means we’re going to be writing to the file in append mode, since we are going to add info to the file over and over every time a user fills out the form. We don’t want to overwrite the previous data, which is what would happen if we used “w” or write mode.

The csv class has a writer class that takes in a file handler, and gives us back an object with methods for adding data to the csv. We name the object writer and use the writerow method, passing in the data the way we want it entered into the row. Imagine this is our csv spreadsheet:

Our csv files can be open in a text editor and look like this:

John Smith, Apley Court
Jane Smith, Matthews

Each column is separated with a comma
and each row is separated with a line break.

Back to our program we see on line 18 this is exactly what the writer.writerow function is doing. Pushing in a row of data from the form.

Finally with use the file.close() method, and return our success.html page.

Our .csv file can be opened in Excel. Create a few entries and try it out!

That is it for week 7 lecture notes!!!

Preview into problem set 6

Coming up we’re going to use flask to create a web app that compares strings char by char, or the less comfortable version is uploading two text files and comparing them in different ways. See you in the next one!

I’m going to show you something I wish I knew sooner.

Python has a program that allows you to setup a virtual environment that is completely walled off from your entire computer. This means you can make a huge mess, write horrible messy code and install a ton of packages that break everything without worry of messing up your computer or other programs. Just exit the environment and delete the folder and you’re done. Keep in mind the folder only holds the environment data so you can write code even outside of your chosen directory.

To do this i’m going to go to my desktop and create a folder called cs50 and then go to that folder in my terminal window.

Make sure you’ve installed python 3 (google download python 3 to get it).

When working with virtual environments with python 3 you have to remember three steps:

• create the env file using python3
• startup the environment using source
• exit the environment with a single command deactivate

So i’m in my cs50 folder in the terminal and i’m ready to create my environment folder. I know it seems confusing, isn’t cs50 the environment folder??? No, we are in cs50 to keep things nice and tidy. More on that in a second…

Let’s run our first command: python3 -m venv .env

So what just happened?

We are using the -m flag which is the command-line flag, to run the venv module as a script. So we’re basically running the venv python program.

the last bit is our argument we pass which is whatever we want to name our folder. I’m going to name it .env to make it a hidden folder (env just stands for environment so I know what it is for).

Now that we have our environment folder ready I can activate it. Still in the cs50 directory i’ll type: source .env/bin/activate

Now my terminal prompt changes so there is a (.env) at the beginning.

I can now install packages and run python3 without worrying about affecting other projects.

Finally if I want to save all my packages I can run this command:

pip freeze > requirements.txt

This creates a text file that has all the packages my project is currently using which I can then install on any other machine by typing:

pip install -r requirements.txt

And that is the way to create a clean directory with a .env file and a requirements file. Run the freeze command anytime you are finished with a project so it is up to date.

If you were putting your whole project into the cs50 folder it’s very convenient to zip it up, send it to someone and have them spin up the same environment from their computer without any issue!

• So let’s save a file in the CS50 IDE, hello.py, with the following contents:print("hello, world")
• Then, we can run python hello.py and see this:$ python hello.py hello, world
  • python is the name of the interpreter program that we’ve installed onto the CS50 IDE, and hello.py is the name of our file that we are passing in as an argument, for it to interpret.

Lecture and Python

The first lecture for the week was done at Yale university and is very hard to follow. It gave me flashbacks of all those days trying to figure out the C problem sets.

Moving on to the next lecture, we’re back at Harvard learning Python. Python is a “productivity” language. It’s easier to use than most other languages and has the added benefit of strict indentation. So your code looks just like my code (indent wise anyways). I have made some pretty basic programs in Python and get the gist of how it works. So this week was yet another breeze! Heading forward to week 7 in record time!

Python Basics (explained in a python program)

Here is a program I’ve written that goes through all the python basics in the lecture:

• If we were to just import cs50 without the from statement, we would have access to cs50 as an object. This means to get to the get_string function we would have to write it as cs50.get_string(“name: “). By using the from statement we have direct access to that function in our program.
  
• using f before a string allows special formatting, for example using variables by wrapping them in curly brackets. Because a string is an object with methods built in, we can also use “mystring {}”.format(myvariable). Using f”string {variable}” is a built in shortcut.
• __name__ is a “special variable” that holds the name of either the main program or the file name where a program is being run. By default the program file I am working in will be “__main__” inside the __name__ variable. If I had a separate file I imported that would have it’s file name associated with the __name__ variable from within itself. Here is a quick example:
  
  create a file called cow.py with the following code:
  

print(f"I am a cow and the __name__ is = {__name__}")

if __name__ == "__main__":
    print("mooooo")
else:
    print("I must be running in the duck file!\n")

Next create another file called duck.py with the following code:

import cow

print(f"I am a duck and the __name__ is = {__name__}")

if __name__ == "__main__":
    print("quaaaack")
else:
    print("I am not a duck")

Running cow.py and then duck.py will show how __name__ is interpreted differently depending on the file being run.

Python Operators

My gist is slightly different from the cs50 notes. Better to re-write it in my own way to really learn more deeply.

The floor division is interesting. It truncates everything after the decimal point. So if you give it 2 integers it returns an integer back. Normal division will return a float.

If at anytime you want to know the type of something, use the type() function which is a part of the standard library of python, which means it just works out of the box!

Python Logic

Logic is very clean in python. Not a lot of braces to deal with. Indentation is important and code will break if things are not indented correctly. Makes looking at code slightly easier (very slightly for me at this point).

Instead of using && and || python uses the actual words or/and which is easier to read, in my opinion.

Using == also works with strings for comparison just like chars.

Defining functions

While looking at the notes on functions everything looked straightforward until I hit this “**” operator. It took a minute to realize python has another operator just for giving you the result of a number to the power of another number. So if I wrote 4**8 I would be saying 4 to the power of 8.

When writing functions we don’t use the function keyword we use def. No curly braces required as long as we indent everything within the function after the colon.

When dealing with longer lines of strings you can break them up as shown above. Just keep in mind if there are variables in your string you have to start each new line with the f character so python knows it’s an “f-string.”

Validate user input

I learned a few things from this example. I always get sloppy when trying to validate user input. In short you have to:

• while true
• if this meets my criteria (if the value is true)
• break out
• return the value

So while true, if true break and return! Input validated.

Also you can create multiline comments in your code by using triple quotation marks like so: ”’

Remember that main() is run at the very end so we can define it at the top and add the other functions later.

Integer overflow

The sleep function is cool for creating a delay. It’s amazing how python has a library for just about everything.

Mario…again

This took a good 30 minutes to sort out in my head. I found the end argument to be super helpful, but don’t forget to explicitly write a print() statement wherever you want an actual line break. By stating end=”” you are saying print this to stdout but don’t include a line break at the end just put nothing at the end. This is also helpful if for some reason every string in your loop needs to have something at the end that is repeated. Maybe a list of names and you just want to say {name}, end=” is awesome!” print().

Command line arguments

If we want access to the program name and any command line arguments when the program is about to run we can import the sys library which is a part of the standard python library. Here is a quick example:

As an added bonus there is a loop that goes through each string argument and then iterates through each character. It is the most readable loop in a loop ever.

Extract a user’s initials

In the cs50 notes for python lecture 8 there is an example for getting a user’s initials from their full name. It looks like this:

# Extracts a user's initials

from cs50 import get_string

s = get_string("Name: ")
initials = ""
for c in s:
    if c.isupper():
        initials += c
print(initials)

I wanted to see if I could figure out a way to get the users initials even if they didn’t capitalize the letters.

This is an opportunity to learn new things, jump into stackoverflow and overcome challenges. Do not feel bad for getting code off stackoverflow! It’s documentation of people spending hours figuring out edge case issues, and if you go there for something in particular over and over you will eventually memorize it. If it is an edge case you are better off leaving it out of your brain to focus on more important things! I did not copy and paste the entire script from someone else. If I got stuck on a small piece I worked through it with help from googling the question.

So I use regex every so often at my job, and it comes in super handy for getting the right bits of text, validating it so you only get what you want. It can be hard to visualize and figure out how to get the right groups.

Regex is a huge topic I won’t discuss here but, I will say check out https://regex101.com/ if you want to learn more.

So here is my initial generator! No capitals required, plus you can have as many names as you want and you will get initials forever!

Linear search in python

We can find an item in either a list of dictionary by using the in operator. It is a straightforward process. I made a quick list and a dictionary to both confirm an item in a list and then bring up an associated price for the item.

Comparing two strings

I created a slightly different version here where I did not use an else branch. sys.exit(0) is a function that will exit out of the program with a code of zero which means program ran successfully. If the string is equal it will pass over that branch and continue to the final print function.

Null and none

get_string can return a value known in python as none. It’s basically null I think!

Swapping values

We touched upon this in my big python program at the top. Packing and unpacking allows us to swap values in one line like this:

x = 1
y = 2

print(f"x is {x}, y is {y}")
x, y = y, x
print(f"x is {x}, y is {y}")

Data structures (classes)

Notice how this gist has 2 files. As long as they are in the same folder from student import Student should work just fine. Creating an object from a class is the same as a function call. We use the class name and the arguments from the __init__ method from inside the class. We don’t need to include self when we create the object because self is a reserved keyword used to bind each argument that it follows. So we’re creating the object and binding name and dorm to that particular instance. So a class is just a template, or a mold for an actual object. When create an instance of the class, we fill the mold and pop out an object.

After gathering the user input in a loop and pushing each one in our list using the append list method, we run a loop printing each student object within our students list.

Saving data to a file in python

This confused me at first because in the example I was under the impression the csv library we import included this weird with open statement. Check this out:

# Demonstrates file I/O

import csv
from cs50 import get_string
from student import Student

# Space for students
students = []

# Prompt for students' names and dorms
for i in range(3):
    name = get_string("name: ")
    dorm = get_string("dorm: ")
    students.append(Student(name, dorm))

with open("students.csv", "w") as file:
    writer = csv.writer(file)
    for student in students:
        writer.writerow((student.name, student.dorm))

So at the bottom we have this new section where we take out students and create a .csv file, which is just a simple spreadsheet table.

with open (“students.csv”, “w”) as file: is all in the box standard python stuff. We are creating a file handler, which I talk about in a previous post.

Without the with statement we have to remember to close our file handler like in C. Using with we can forget about it, python takes care of closing the file handler for us.

Our fancy csv library comes with a method called writer that takes in a file handler and gives us a writer object. That object has a method called writerow that takes in a tuple which we wrap in parenthesis and writes that data one row at a time into our file. In my example I push column headers into the first row before the loop starts.

What a dictionary class looks like in python

Almost done with the week 6 notes and i’m hit with this dictionary class! So lets try to break this all down:

class Dictionary:
    """Implements a dictionary's functionality"""

    def __init__(self):
        self.words = set()

    def check(self, word):
        """Return true if word is in dictionary else false"""
        return word.lower() in self.words

    def load(self, dictionary):
        """Load dictionary into memory, returning true if successful else false"""
        file = open(dictionary, "r")
        for line in file:
            self.words.add(line.rstrip("\n"))
        file.close()
        return True

    def size(self):
        """Returns number of words in dictionary if loaded else 0 if not yet loaded"""
        return len(self.words)

    def unload(self):
        """Unloads dictionary from memory, returning true if successful else false"""
        return True

I’m going to have to go back and get the list of dictionary words from that challenge way back when. I will create a gist with the file. Back in a few…

Week 4’s speller walkthrough! So I found the txt of words to load into our dictionary. So let’s see this in action:

from dictionary import Dictionary

dictionary = Dictionary()

print(dictionary.load('large'))

print(dictionary.size())

sample_text = []

with open('aca.txt', 'r') as file:
    for line in file:
        line = line.split()
        for word in line:
            if word[0].isalpha():
                sample_text.append(word.strip())


with open('spell_check.txt', 'w') as file:
    for word in sample_text:
        if not dictionary.check(word):
            file.write(word + '\n')
            print(f'{word} not in dictionary!')

So you need the week for files which i’ve included a link to here. This gives you a zip file with all the files from that challenge.

This is nothing special and barely does what it should do. BUT the actual “correct” program was barely doing a good job spell checking so i’m ready to move on!

Fortunately for me I have had prior experience in full stack web development, after taking a few classes over at teamtreehouse.

I think that is probably why the open source degree program i’m taking suggests I should move onto the next course at this point.

Although I agree, I want the verified certificate once I’ve completed the course. Which I will add to my “badges” page (coming soon!).

Questions?

If you are taking this course and made it to week 5, you probably understand how the internet works. If you don’t that’s ok too, and I would like to help you understand if there was something in the lecture or the video shorts that did not make sense. Feel free to leave a comment with your question and I’ll update this post with a detailed answer.

On ward to week 6!

I’m going to do my best to explain the solution in small, manageable steps so you can follow along and not go through the pain and suffering I did!

Maybe running a gauntlet is harsh, but some days were brutal, and at one point I even deleted tons of code after being completely fed up.

If this happens just take a break. Work 30min to 45 min at a time and go for walks or just stand up for 5 to 10 min in between. Make small pieces of programs that prove a belief about what you think the code is doing.

The last C problem set is called speller and is simple when you explain it at a super high level.

Main Goals:

• load() the dictionary word by word into a data structure
• Keep track of each word added and return the total number of words in our dictionary by implementing a size() function
• We are given a single word from the passage in a loop we do not have to implement ourselves. Looping through each word in the passage and pulling it has been done for us already! We only have to take a single word and compare it against our data structure within the check() function
• Finally we have to loop through each index of our data structure (it’s just an array folks) and free all the dynamically allocated memory (this is not as hard as it sounds). This happens within our unload() function

It processes a piece of text, looks at each word against a dictionary of words and returns any words not found in the dictionary as “misspelled.”

Yeah…simple right?

Implementations

So before I show my implementations, let’s go through each function. Remember a lot of this program is already done for us! Also the program puts each function in alphabetical order instead of the order of when they are called within the program. I will explain each in the order they are called from speller.c.

We need to know what our functions are expected to do, and the only way to know is to check what is being passed to us by the main program!

Use GDB! Printf is great for little things but when you’re really lost in a bigger program gdb is printf on steroids! (click on the text to go back to the article where I walk you through GDB)

bool load(const char* dictionary) part 1

The argument passed is a char* (a “string”). The const part means the value can not be changed. If we tried to change the string we are passed we will get an error. We should return the value true o

Let’s use GDB to see what is going on in the main program speller. We’ll create a break point right on the line load() is called, and step into the function so see what is being passed. In my speller.c file the call to load() happens on line 45.

Remember we start gdb by typing gdb and the filename without the .c part. Now GDB will startup and give us a prompt “(gdb).”

So this is all just preparation before we run the program. First we have to tell gdb where to pause in our program. I’ve typed in b 45, which means when I hit run go through the program and pause, or in other words break at line 45.

Now we just have to type the letter r to run the program:

Usage: speller [dictionary] text gives us the hint we need. When we typed in r gdb ran the program as if we typed ./speller but the program has 2 arguments we need to pass in the terminal.

The first [dictionary] is optional, which is why it’s surrounded by square brackets. The second argument text is mandatory, which is why we couldn’t run it.

Check out the folders that are within pset5 using the command ls once you cd into pset5 from the ~/workspace/ directory.

You will see texts/ and dictionaries/

[dictionary] is optional because our main program will automatically choose the large file within that folder.

text has to be choosen. I did some exploring in the text folder and found a small piece of text called quote.txt:

It is known that there are an infinite number of worlds, simply because there is an infinite amount of space for them to be in. However, not every one of them is inhabited. Therefore, there must be a finite number of inhabited worlds. Any finite number divided by infinity is as near to nothing as makes no odds, so the average population of all planets in the universe can be said to be zero. From this it follows that the population of the whole universe is zero, and that all people that you may meet from time to time are merely the products of a deranged imagination.

-Douglas Adams – The Restaurant at the End of the Universe

So what we really need to type is r texts/quote.txt. Let’s try again:

So we’re in speller.c on line 45 (speller.c:45) and the program is loading in something called dictionary. So the question is what is dictionary? Before we step into the function let’s just check using the p command which is short for print.

So instead of typing printf and trying to guess the value we can just use the command p and any variable of any type a

So back to our output p dictionary:

So it passes the string “dictionaries/large.”

From the lecture and shorts videos, we know that loading the dictionary means creating a data structure. This is a super fancy word. Don’t be fooled into thinking a data structure is some super complex thing only a genius can create. It’s just an array, that is all…

An array of structs…

connected to other structs known as a linked list…

which we insert using a hash function…

oh… and the array is a global variable so we can access it within other functions…

OKAY MAYBE IT’S A LITTLE COMPLICATED.

Let’s get out of GDB for a moment, open dictionary.c and start coding within the load() function.

(to quit out of GDB just type q for quit. It will ask if you are sure. Just type y for yes.)

First we need to use the filename we’re passed (const char* dictionary) and use it to open the file.

Now paste this code into the load() function and run the make command:

    // create a FILE pointer and fopen() to start the stream
    FILE * dictionary_stream = fopen(dictionary, "r");

    // print out the first character
    printf("first character of the file is: %c\n", fgetc(dictionary_stream));

    // use GDB to inspect the test_string variable after it is used each time.
    char test_string[LENGTH+1];

    // fgets will give us everything on the current line 
    // and put into test_string
    fgets(test_string, (LENGTH+1), dictionary_stream);

    // fscanf gives us whatever we say in the format argument (like %s)
    // and put into test_string
    fscanf(dictionary_stream, "%s", test_string);

    printf("end of example\n");

run GDB again using the same commands we did before. Here is a quick list:

• gdb speller
• b 45
• r texts/quote.txt

At this point when we run quote.txt we get:

45          bool loaded = load(dictionary);

load(dictionary) is about to run.

It has not yet run, but if I type n and go to line 46 (the next line), load() will run.

Everything that load() needs to do will be over and passed to the variable bool loaded.

I don’t want that.

What I want is to step

For this we have to type s for step-into.

now we’re inside the function and get this as our output in GDB:

FILE * dictionary_stream = fopen(dictionary, "r");

We can now type the n (stands for “next”) command to see what our variables are doing in the next few lines.

type n enter

then type n enter one more time. 

Your output should be something like this:

first character of the file is: a
158         fgets(test_string, (LENGTH+1), dictionary_stream);

Ignore the fgets part for now…it won’t run that line until we hit the n key.

Here we can see our first output, which is the printf we added to our function. It reads “first character of the file is: a.”

So our dictionary_stream was at the beginning of the file, (think of a play head on a youtube video) we used fgetc() which collected the first character, which then in turn moved the FILE pointer dictionary_stream to the next character.

You don’t have to open dictionaries/large, I will tell you the first few lines are as follows:

a
aaa
aaas
aachen
aalborg

So fget(dictionary_stream) grabbed the first a a

But wait a minute…

Did it really move the pointer to the next newline?

Let’s move ahead and see…

hit the n key again and press enter.

fgets(test_string, (LENGTH+1), dictionary_stream);
(gdb) n
161         fscanf(dictionary_stream, "%s", test_string);

So the first line above is the same one I told you to ignore a moment ago. Once we hit n and the enter key it brought us to fscanf. Which has not yet run so we can’t look at it’s value yet.

So let’s see what test_string is holding after running the fgets function. If you did not know fgets() takes everything from a single line and puts it in the char* variable we specify, in this case test_string.

run p test_string

$1 = "\n\000@", '\000' , "\220\336\377\377\377\177\000\000\360\334\377\377\377\177\000\000\000\000\000\000\000\000\000\000\200N\335\367\377\177"

Your garbage values may vary but the important part is near the beginning of the output. Disregard the $1 = “ at the start.

Why did our function grab \n first?

Look at the text at the beginning of our dictionaries/large again and see if you get why:

a\n <- this \n is on every line but is invisible in the text editor
aaa
aaas
aachen
aalborg

It’s because our first function fgetc() only grabbed the letter, not the invisible character \n that represents a line-break.

If we never ran the first function fgetc() and only ran fgets() we would have gotten this output instead:

$1 = "a\n\000@", '\000' , "\220\336\377\377\377\177\000\000\360\334\377\377\377\177\000\000\000\000\000\000\000\000\000\000\200N\335\367\377\177"

So now we know running fgets() will grab everything from the current file pointer position up to and including the \n (line-break special character).

Continue on by typing n a

161         fscanf(dictionary_stream, "%s", test_string);
(gdb) n
164         fscanf(dictionary_stream, "%s", test_string);

So again the first line we already saw, but since we’ve move one more line ahead we can now see what test_string is holding after running fscanf.

type p test_string. The output should be something like this:

$1 = "aaa", '\000' , "\220\336\377\377\377\177\000\000\360\334\377\377\377\177\000\000\000\000\000\000\000\000\000\000\200N\335\367\377\177"

Again the garbage values are not important and will most likely be different for you. But the first part is the most interesting:

"aaa", '\000'

Since the previous function grabbed the line-break. The pointer moved to the start of the next line. If you scroll up and check the second line of dictionaries/large was in fact the string “aaa.”

Let’s run the next fscanf()

type n

type p test_string

now the output should be something like this:

166         printf("end of example\n");
(gdb) p test_string 
$2 = "aaas", '\000' , "\220\336\377\377\377\177\000\000\360\334\377\377\377\177\000\000\000\000\000\000\000\000\000\000\200N\335\367\377\177"

BINGO. Ignoring all the garbage values and previous printf statement we can clearly see “aaas” followed by the null terminator and some more zeros.

!IMPORTANT

See all those weird garbage values? That’s because when we declared our variable test_string[LENGTH+1], We didn’t initialize it to anything. Here is a trick I learned after I couldn’t get my code to work and went on a google binge for a few hours:

Go to the line char test_string[LENGTH+1];

and change it to char test_string[LENGTH+1] = {‘\0’};

Quit out of gdb, run make and then go back into GDB and step into the function and p the values by moving through our load()

Now you’ll get output similar to this:

$1 = "aaa", '\000' <repeats 42 times>

When you initialize the first index, it automatically makes all other indicies a null terminator.

This might not matter now, but when comparing strings we will have to worry about every index of our array (in my implementation anyways).

bool load(const char* dictionary) part 2

Now erase all the test code we’ve written in our load() function.

We know fscanf is the way to go for reading in each word, so let’s talk about it’s technical definition and how we can use it in a loop to get all our words and start creating our data structure.

From tutorials point:

Description

The C library function int fscanf(FILE *stream, const char *format, …) reads formatted input from a stream.

Return Value

This function returns the number of input items successfully matched and assigned, which can be fewer than provided for, or even zero in the event of an early matching failure.

So we’re given the filename from the main function. We still need that dictionary_stream pointer to the file so let’s put that back into the function:

// create a FILE pointer and fopen() to start the stream
    FILE * dictionary_stream = fopen(dictionary, "r");

We know there is a #define in the main function where LENGTH is a constant for the longest word possible. Let’s also create a place to hold the current word in the loop:

char temp[LENGTH+1] = {'\0'};

Remember we need that +1 for the ‘\0’ null terminator if we ever come across a word that is the max length.

So now fscanf() will return the number of things we tell it to pull on success. So what are we asking it for? The second argument we pass it is the format specifier. 

returns an int: fscanf(the file pointer, format you want, a place to store the thing you want)

So if we say hey fscanf here is the dictionary_stream I want you to pull a %s which is a string, and store it in temp, 

Because the file pointer dictionary_stream moves automatically everytime it’s called we just have to keep running fscanf() until it no longer returns the number we are expecting.

So each time we call fscanf it pulls in a single word. We know this because we specify “%s” which is a one string. So by that logic we can say while fscanf returns the number 1 keep calling the function.

Let’s create a small test to see if this is working. Here is the entire function as it stands to avoid confusion:

If we run ./speller texts/quote.txt while in the pset5 directory we get this output:

143091
Could not load dictionaries/large.

Since we’re returning false the main program returns early with the response “Could not load dictionaries/large. So hopefully you understand what the number 143091 represents.

We’ve successfully pulled in each word in our loop! We aren’t currently doing anything with that variable temp which holds the current word in that loop, but we now have the total number of words available within the counter variable.

A Quick Detour

Before we continue loading each temp variable in the while loop into a data structure, let’s score any easy win by implementing another function using only a few lines of code. We have that counter v

Within dictionary.c we have another function just below the load() function called size().

Above the function the comment reads: “Returns number of words in dictionary if loaded else 0 if not yet loaded.”

Now if you didn’t already know, variables declared within a function only exist in that function.

Go to https://repl.it and spin up a quick C programming environment to run this code for a second:

#include <stdio.h>

char *global_animal = "bears";

char *lions(void){
  printf("You can not escape the %s!\n", global_animal);
  char *animal = "lions";
  return animal;
}

char *tigers(void){
  // the variable animal is created
  // within lions but this function
  // has no idea.

  // try to run the code with the
  // line below uncommented and you
  // will get an error
  //printf("NOPE CITY %s\n", animal);


  printf("%s are everywhere!\n", global_animal);
  char *animal = "tigers";
  return animal;
}

int main(void){
  printf("%s, %s, %s, oh my!\n", lions(), tigers(), global_animal);
}

So given this information we know we have a counter variable in the load() function where it is building up the final count of words. But we need to pass it along to the size() function.

Let’s declare the variable outside of the load() function at the top near our #include directives.

#include <stdio.h>
#include <stdbool.h>
#include <stdlib.h>

// global variable
int dictionary_word_counter = 0;

I’ve changed the name from counter to dictionary_word_counter. Go back to the function load() and remember to change the name there too!

Now our load() f

All we have to do for size() n

/**
 * Returns number of words in dictionary if loaded else 0 if not yet loaded.
 */
unsigned int size(void)
{
    return dictionary_word_counter;
}

Some cats…

bool load(const char* dictionary) part 3: The Hashtable explained

Getting values from an array is easy when you know the index. If I have an array called numbers[42, 18, 9001] and I want 42, it’s easy if I know to look in array location [0] right? But if I don’t know where my number is I need to run a loop and check each array location until I get a matching number and then return that index.

Running that loop on 3 items isn’t a big deal.

Running that loop on 143,000+ words is a big deal.

It will either take a very long time, or crash. So we have to find a way to place a word in the array quickly and get it back quickly and it has to work exactly the same.

The good news is some smarty pants person already found a solution for this, we just have to implement the process.

This process of fast insertion and retrieval is called a hashtable and it involves taking

• a regular array

• our regular char* text

• a funky function we’ll called generate_hash

generate_hash() takes our text and calculates our index based on some math it will perform on the characters. Remember characters have ascii codes that are integers so we can do some clever mathematics like adding up each word and using the output to determine the index we will place it. DO NOT IMPLEMENT YOUR OWN HASH FUNCTION. It’s a big to do that this course doesn’t cover. Here is a link: 

https://stackoverflow.com/questions/628790/have-a-good-hash-function-for-a-c-hash-table

Confused? Here is a drawing:

So now that you understand how we can place words in our data structure we need to take things one step further, using something called a singly linked list or linked list for short. We can not hold 143,000 array indexes because our resources on the stack are limited. We have to hold a pointer to a linked list and search for our word. I’ll do my best to explain all this.

The Stack and The Heap

We put everything we declare in our program on the stack. Anything dynamically allocated at runtime using malloc() we put on the heap. Unfortunately for us the stack won’t hold 143,000 array indexes. We need to lower the number of indexes and somehow use the heap.

Imagine the stack as this nice orderly piece of data like the array in the picture above, while the heap is a mess of chaos. We have to get clever to leverage the space the heap gives us without losing the stuff we throw in there. Try to use this image as a reference we’ll talk about later:

The heap is massive in comparison to the stack but it’s complete chaos. Using malloc() we can utilize a portion of heap memory. When we run the memory-allocation function it returns the address to the first block of memory since we can allocate multiple blocks.  We can allocate 4 blocks of size(char) for example.

The biggest challenge when dealing with heap memory is that we only get an small address. Confused what that means?

The address to your house is a few words and numbers. That can lead a person to your actual house where you have your computer, toaster, clothes and other house hold items. If someone is 50 miles away and they have no idea where you live, and no address they won’t find you.

The address to a soccer stadium is still just a few words and numbers but can hold thousands of people. Lose this address, you will miss the soccer game.

Because we are given a pointer to the memory address, the stack can handle that tiny piece of information. We can leave the heap to deal with the actual values. We always have to keep track of the addresses. Keep this all in mind as we talk about building a linked list.

Singly Linked Lists

Run this little c program:

#include <stdio.h>
#include <stdlib.h>

int main(void) {
  int *one_char = malloc(sizeof(char));
  int *two_char = malloc(sizeof(char)*2);

  *(one_char + 0) = 'a'; // same as *one_char = 'a';

  *(two_char + 0) = 'b'; // same as two_char = 'b';

  *(two_char+1) = 'c';

  printf("%c, %c, %c\n", *one_char, *two_char, *(two_char+1));

  return 0;
}

Here is a diagram of what is happening in our little program:

First we malloc enough space for one character. This creates the first red circle.

Then we create the second bigger red circle to hold 2 blocks, each the size of 1 character.

Then we assign a value to that block of memory. We can use pointer arithmetic to access different blocks of memory. I’ve put a number below each character to show the corresponding number you have to add to the dereferenced pointer.

Notice how our stack only has to hold a small memory address that points to the chunk of memory. Is it possible then to string along multiple chunks in one array index on the stack? Let’s make a struct in C and call it a node, then let’s make a tiny linked list:

Check this diagram to visualize the code above and try to figure out what is happening:

bool load(const char* dictionary) part 4: The finale

So here are all the things our load() function should do:

1. Access a global FILE * dictionary_stream and fopen() the filename passed in from the main speller program
2. Declare and initialize a char array of size length + 1 for the NULL terminator
3. Create a while loop while fscanf() returns the number 1
   1. within the loop create space on the heap for our struct node (do not get confused with node_ptr we need room for a node but we are passed a node_ptr to store on the stack which points to our newly acquired space on the heap.)
   2. save the current word in the loop to the node’s word property
   3. generate the correct index (generate a hash) to place the node_ptr
   4. when inserting into the array make sure to check if there is an item at that array location, which means we have to do some singly list rearrangement techniques, for example placing the new item to the “head” of the list. Just keep in mind we need to create a temp node_ptr that knows the address of the old head of the list (the original array value on the stack). We update the array index value to the newest address to the newest node, Then we take the newest node’s next property and assign it to the temp node_ptr. Which means we’ve weaved another item to that array index. The diagram above is an example of what it should look like at the end.

Hopefully if you piece together all the ideas in these little programs i’ve shared, plus some creative programing you can come up with your own solution. Here is what I came up with:

Do you want more explanations?

unload() loops through each array location and free’s each node from the linked list. I created a function free_linked_list() to make it look cleaner.

check() takes in a word from the passage text and after “cleaning” (all lower case) the data, generates a hash number (an index in the array to search), and loops through however many nodes exist at that location, comparing each dictionary word to the given passage word. We just have to return true or false.

You will see at the top of my solution I created functions for working with nodes and generating the hash. If you want me to go more in depth on any of those other functions please leave a comment and i’ll do my best to share some more diagrams and clarify what is happening. My goal in this article was to try and simplify the idea of data structures and sharing the solutions to some of the most frustrating moments I had while trying to write up a working program.

As always,

rest, but never quit.

New format…

Please comment if you have any questions and i’ll do my best to respond with a detailed answer to help clear anything up. I’m noticing these problems went from wow that was difficult, to wow what is the meaning of life how could anyone solve these problems in a week?!

Padding question

The readme had me stumped on this one:

What value does line 65 of `copy.c` assign to `padding` 
if `bi.biWidth` is `3`?

Here is the code in question:

// determine padding for scanlines
int padding = (4 - (bi.biWidth * sizeof(RGBTRIPLE)) % 4) % 4;

So broken down:

We're given 3 for the width and a rgb triple is 3 as well:

int padding = (4 - (3 * 3) % 4) % 4;

// 3 * 3 = 9

int padding = (4 - (9) % 4) % 4;

// 9 % 4 = 1

int padding = (4 - 1) % 4;

// 4 - 1 = 3

int padding = 3 % 4;

// 3 % 4 = 3

int padding = 3;

Does modulo still confuse you? Whenever I step away for a couple of days I lose my “modulo mojo.”

Why does 3 % 4 = 3

because mod gives us the remainder of a division operation.

Look at the expression 8 % 4 for example.

Imagine the 4 hits the 8 and gives 4 damage on the 8. So that 4 can hit 8 twice before there is nothing left.

If the first number was 9 there would be 1 left. That’s what mod gives us, whatever is left after the second number hits the first number.

So what about 3 % 4?

Well 4 can’t even fight 3 because it can’t get at least 4 damage out of it. So the fight never happens and modulo gives us what is left in the first number which is the same as what we started with which is 3!

Whodunit, struct and typedef explained

Although it can be uncomfortable at first to wade through someone else’s file of code, this was a little easier to solve than last weeks helper functions. Remember the lecture speaks about file pointers and modeling our data using structs.

Here is a simple program to help demystify things:

View the code on Gist.

I’ll show my answer and explain what is going on (starts on line 80):

View the code on Gist.

In our loop we’re streaming in one pixel of RGB at a time. To access the value of the current pixel we can dive into our struct’s properties. To access a property we take the struct, which in this care is triple, add a dot and then our property:

triple.rgbtRed

So we can print this value out or run a test. I wanted to remove all red pixels so I created a branch within the loop between the fread and fwrite functions. This is the perfect place to make modifications because it’s right before we write into the new file!

I decided my test would be if a pixel had any full red value. In hex thats 0x0000ff.

If it did I would take that pixel and black it out on all channels. We can do that by taking each property and assigning it the value of 0x000000, which is black.

GDB to the rescue

Working through resize more less comfortable right now. Figured I would give the more challenging problem a try. After watching the video i’m faced with resize.c and bmp.h feeling very confused as to where I should start.

Try this:

1. make your copy of the file copy.c (I named mine resize.c) and use make
2. run the command gdb resize
3. set a break point after the header variables are created and read into (this was around line 45) using the command b 45
4. run the command r <infile.bmp> <outfile.bmp> (using your own bmp files)
5. now take a look at where you are in the code by using the command list
6. finally print out either the bf or bi structs using the command print bf or print bi

Now you can see the insides of those variables and what they are made up of when read. Try printing bf.bfSize etc. It makes a huge difference in my comfort level and I didn’t have to write a single print statement!

Understanding File I/O in C

File I/O operations that occur in resize less comfortable. We have structs being created from a header file, some pointers getting thrown around and what the heck is that nested for loop monster doing at the bottom? Weren’t we support to write neat little functions for everything?

Once you understand what is going on in each of these simple programs, write it from scratch yourself. Change the text, add a conditional if you want to be adventurous. Maybe if somewhere there is the character ‘s’ switch it to ‘z’. Make it your own to really learn what is going on. Each time you run make and there is an error, solve it and be amazed at your growth as a programmer!

Creating a new file:

View the code on Gist.

Reading in a file:

View the code on Gist.

Visualize file I/O in C

FILE * fp = fopen("stuff.txt", "r");

The FILE data type is known as an opaque data type so the details are hidden. You should also know FILE is a struct which means this creates an object. For now we need to know this is the data type when dealing with files. Simple enough for now…

Next we have the * fp. So we know that fp is a pointer which holds the address where our file lives. But what if we don’t have a file yet? Well we don’t quite have access to an address space but a “stream.” The fp becomes a “file handler.”

We take for granted how simple it is to save files on our computer. We don’t think for a second about it. Back in the day this was an issue. Programs store data to memory (RAM) while they’re running, and if there is enough memory, but once stopped (or shut off from power) programs lose everything stored on the computers memory. In order to save data after programs stopped they needed a completely different way of storing data. Files became the way, and needed a way to be HANDLED.

Memory (RAM) is usually a long chip with squares. File storage devices are not all alike. DVD’s, SD cards,HDD’s and SSD’s are all storage devices that you find in modern computers today. But it did not start out that way at all!

In the past, data was stored on giant spinning disk machines. There were programmers that had to implement code that could save data to these.

Let’s create a pseudo function for that:

saveDataOnSpinningDisk(){
  // lots of programming
}

It’s 1965 and your manager comes in and tells you “They have these new tape machines that store 100 times the data in the same amount of space! we have to write code to save data to these new data storage machines!”

All our programs (think thousands of lines of code) have a function called saveDataOnSpinningDisk().

So what do we do? Maybe we have an if statement like this:

if(using old data storage method){
   saveDataOnSpinningDisk()
}else{
   saveDataOnNewTapeMachine()
}

A few years later the HDD (Hard Drive Disk) is invented, and a basement full of tape machines can be stored in the palm of your hand. Management is telling you to write more code to HANDLE FILES using this new method of storage. Soon CD’s DVD’s Blue-ray discs, SD cards…you get the picture.

Eventually people figured out data storage could be abstracted out to their basic operations. All of these storage devices can be written, read, appended to, deleted, and created. By keeping the specific implementation behind the scenes, our programs only have to deal with an abstraction called a FILE HANDLER, that handles these I/O operations for us.

The C programming language is taking care of the file I/O specific to our system for us. It knows we’re on linux ubuntu, on the CS50 cloud server and does the file I/O needed for the system. Many digital cameras use C, which means the same functions you write to manipulate file I/O can be used to store data on an SD card. This keeps you focused on the program at hand, not the distracting details of file I/O.

I finally got through week 3

Honestly,  it didn’t take 8 weeks. It took probably a week of really focused time to complete this problem. Specifically “less comfortable.”

I’ve created a few simple programs that helped me understand each part of the program resize.c, so it wouldn’t feel so overwhelming when trying to tackle in one pass.

Not sure how to create a file or make a copy? This little program might help:

View the code on Gist.

Not sure what the original copy.c is doing in the problem set? Try going through my version and see if it makes more sense. I added a lot of helpful comments.

View the code on Gist.

Things in less comfortable that sucked

I made many attempts but ultimately went on GitHub to find other programers solutions. My end goal was to understand the code. Reading it line by line I was able to finally understand what had to be done to complete the challenge. I’ll put my solution below and we’ll go through each section:

View the code on Gist.

Validating inputs

Up to line 59 we’re just assigning the user inputs to clearly named variables and making sure our file streams are valid and working. On line 22 we have to use the enlargement factor argument as an int data type so using atoi().

The part on line 51 about making sure it’s a 24-bit uncompressed BMP 4.0 is a detail we don’t have to worry about and are just copying from copy.c.

Update the headers

This covers all the way to line 87. We’ve got 2 headers, the BITMAPFILEHEADER and the BITMAPINFOHEADER. Read and write in the correct order! I kept bmp.h open in another tab in the cs50 ide to use as a reference on how the structs are ordered and what data each is holding.

It’s best to keep the infile data and the outfile data separate so I created variables for the infile headers (bf and bi) and variables for the outfile headers (obf and obi).

So the video reading less comfortable tells us all the things we have to update which are:

• width & height
• biSizeImage
• bfSize

Getting the values in this order is key because each is dependent on the former being updated.

Once all the updates are complete we use fwrite() to push that information into our outfile.

After each read and write operation, the infile pointer and outfile pointer is moving forward. Imagine a youtube player and when we read or write the little playhead ball is moving ahead.

The difficult part

Starting from line 91 we get caught up in a lot of nested loops.

First we have to think (whiteboards are very helpful here) about what we’re doing to the RGBTRIPLE in the loop. Let’s draw it out:

before a resize with a resize factor of 2:

[] []~

[] []~

after a resize:

[] [] [] []~

[] [] [] []~

[] [] [] []~

[] [] [] []~

([] represents one RGBTRIPLE)

(~ is padding and the values can be different)

When we read in a single RGBTRIPLE, we have to write it out n times. Whatever factor the user gives us. The tough part comes into play when we try to duplicate the rows.

Notice how the after drawing has 4 rows instead of 2?

I’ve seen some solutions using fseek by factoring out the 54 bytes of the header and looping through, but it was confusing for me so I choose to use the malloc() solution.

We store the entire current row of RGBTRIPLE’s (pixels) into scanline AFTER we duplicate the number of pixels using a nested loop.

Then push them n times into our outfile.

So the inner loop collects what our new row looks like and the outer loop actually writes to the outfile an entire row at a time.

before:

[][]

after:

[][][][]

scanline takes in an entire row [][][][]

then pushes the row into the outfile in the nested loop plus padding factor  times

Back up more to line 91. We have a data type RGBTRIPLE but it’s a pointer type and we’re using malloc(). I was hesitant to use this for about 2 weeks.

Best way to learn is to practice with little programs one without malloc() and one with malloc():

View the code on Gist.

So the above program is without malloc. We declare an array of chars size 3. We know the file is 3Max, 3 being our version of a “header” telling us the “width” is 3, or 3 chars. If we change the file 3Max to 3JohnnyYzaguirre we aren’t going to get the result we want because our variable holds a static number of chars.

We need something dynamic, which means something that can size itself  based on what is required.

Malloc() allocates memory by supplying it with a number of bytes. We assign this to a pointer of whatever data type we want to store.

By having the pointer of a data type (int, char, RGBTRIPLE etc), we can use pointer arithmetic, which allows us to move around the blocks of <datatype> memory very easily that we have allocated. *WARNING THIS IS NOT LIKE A FILE STREAM! THE POINTER DOES NOT MOVE ON TO THE NEXT AVAILABLE PIECE OF MEMORY WHEN YOU READ OR WRITE FROM IT!

Try out this program to see what I mean. Lots of helpful comments here:

View the code on Gist.

resize less comfortable throws a lot at you so i’ll give you a few hints that I don’t feel I completely covered in the article.

• learn to use fseek() by looking up the function arguments. It’s like a functional way of taking the playhead backwards or forwards. You only need to push it forwards in this challenge. Once you have the correct outpadding you are going to manually add it after you’ve pushed a scanline into the outfile.
• More on the loop section: The main loop represents the rows of the infile and has 2 inner for loops. the first inner loop represents the individual pixels in a row. Just like in our example with Max, we take that temp pixel variable and write it over and over factor times using a nested for loop. the second inner loop is where we actual write the scanline to the outptr followed by another nested loop where the padding is manually added.
• The scanline pointer_counter lives inside of the main loop. This is because we actually need it to reset to zero since we’re re-using the scanline for the next row from the infile.

Happy coding everyone! Until next time =)

I’m going to be working on my blog posts while doing the walkthroughs next time around. It’s hard to remember what exactly stumped me and how I got around it otherwise. Here are my solutions for the Week 2 walkthroughs. This project was difficult and frustrating for me. I found using the CS50 debugger was super super helpful in figuring out where I was going wrong. Use it to follow the values of variables throughout the program.

View the code on Gist.

Leave a comment if you have any questions and i’ll do my best to answer them!