Prompting Progress: The Generative AI & Prompting Course at UCalgary

Last year (2023) around this time, the generative AI revolution had begun, and I felt like the students at the University of Calgary were just as hungry to learn about this technology as I was. I had already started incorporating a lot of generative AI content in the courses I taught about technologies of entrepreneurship and innovation at the Haskayne School of Business, but students needed more, and not just students in the business school. So I took the plunge and committed to offer a full course on "Generative AI and Prompting" available to both undergraduate and graduate students from across campus. The University's newsletter promoted the course, and it also got some news coverage. You can listen to my CBC radio interview about it here.

In this post I want to highlight some aspects of the course in case it is helpful to prospective students thinking about taking the course or other instructors teaching similar courses. The new offering of the course starts in September 2024, Tuesdays and Thursdays at 9:30-10:45 AM. The course code has since changed to ENTI 333 for undergrads and ENTI 633 for graduate students.

The Generative AI Discovery Forum

Given that generative AI was new to all of us, I framed the learning experience as a collective discovery process, and wrote about it here. We were all on a shared journey of exploration and learning. A key component of the course that enabled collaborative discovery was an online discovery forum. Students were encouraged to contribute a wide range of discoveries to the forum throughout the course, including:

1. Newly released AI applications or tools
2. Valuable learning resources such as books, articles, websites, Discord groups, or Twitter threads
3. Novel prompting techniques or templates found online
4. Personal insights into AI behavior or innovative use cases

The forum's structure promoted active engagement, with grades assigned not only for posting but also for voting and commenting on peers' contributions. This approach incentivized students to stay current with the latest developments and critically engage with their classmates' findings. Moreover, the discovery forum was a useful resource for me as the instructor, providing a stream of content that I could learn from and could incorporate into class presentations and discussions.

AI-Open Quizzes

The course featured brief multiple-choice quizzes spread throughout the term, but with a twist: students were permitted and even encouraged to use any resource or tool to assist them, including AI. I tried to craft AI-resistant questions that even ChatGPT couldn't consistently answer correctly. However, this task is becoming increasingly challenging as AI models rapidly advance in capability.

With strict time constraints (approximately one minute per question), students faced strategic decisions about whether to employ AI tools, consult other resources, or rely on their own knowledge. These quizzes weren't solely about answering questions; they served as a practical exercise in discerning when and how to effectively utilize AI tools for such tasks in time-sensitive situations.

Post-quiz discussions allowed students to share their experiences and strategies, fostering peer learning. For instance, some students discovered they could feed audio recordings of class sessions to AI tools for transcription and content retrieval via text queries. As AI models get more capable, I am still debating about whether or not I should retain this form of assessment.

Prompt Engineering

A prompt is the input you give an AI model or tool to get a desired output. When you go on ChatGPT you can just type anything and very often you get good results with just a simple intuitive prompt. Because it's so easy, it can be hard to appreciate how much effort goes into the best prompts. I like to define Prompt engineering as the use of special techniques in prompting to increase its effectiveness. To really drive home the point that prompt engineering is a serious skill and there is both an art and a science behind it, one of the key assignments in the course is a prompt engineering report where students have to craft a single really good prompt, and report on how it works, why it works, what problem it solves, variations of the prompt, and ethical considerations for use of the prompt.

The Prompt Crafting Framework

To help students appreciate what makes a well-engineered prompt distinct from trivial prompts, I have developed a prompt crafting framework, that I continue to update and revise. As of the time of this writing, the current version my prompt crafting framework is this:

1. Non-Triviality: These prompts are not obvious or trivial. They are not something that an average person would typically think of writing as a prompt without any external resource or template.
2. Machine psychology: Prompts that take advantage of the particularities and quirks of how the AI model “thinks,” especially when these quirks are counter-intuitive, provide value. For example, researchers have found that generative AI performance can be improved through prompting that is motivational, encourages reflection and self monitoring, considering alternatives, and step by step thinking through a problem. Sometimes generative AI models work great with prompts that are not purely in human language, and use a combination of code-style instructions with natural language.
3. Expert Insight & Priming: They go deeper into areas of expertise or bring in more detailed expert or domain knowledge than the average person would with a simple prompt. For example, instead of asking Midjourney to create interior design mock-ups, you would specify a specific interior designer's style and keywords related to that style, or describe an interior design context in such detail that the AI can create mock-ups useful for that context. This typically involves reviewing important research and incorporating the findings of that research into the prompt. An important impact of this expert knowledge is that the more expert language used in the prompt provides the AI with context priming, guiding it to produce more expert-level output because the statistical patterns in its neural network are more likely to associate expert output with expert input.
4. Expert Detailing: They specify details in ways that require expert knowledge or a considerable amount of time and effort to write (for example styles of important artists known by experts, or step by step instructions or valuable guidelines provided by experts, or lists of keywords that require time and research to collect).
5. Framework Application: They apply helpful frameworks of thinking and reasoning to aid AI-assisted thinking (for example design thinking, reasoning techniques, two by two frameworks).
6. Creativity & Novelty: They connect ideas or styles together that were otherwise difficult or costly to connect or others hadn’t thought of connecting them before or hadn’t realized the value of connecting them.
7. Discovery & Insight: They help people discover things they did not know or identify their blind spots.
8. Access Expansion: They allow us to benefit from the ideas, knowledge, or styles of writing or thinking of people we would otherwise not have access to or would be difficult to access.
9. Learning & Problem Solving: They help people learn concepts or material that would otherwise be hard to learn or solve problems that are difficult to solve.
10. Representation & Inclusivity: They allow us to foreground the contributions of marginalized or lesser-known experts or enable us to learn more about marginalized, oppressed or silenced people(s), or highlight such people(s) in outputs despite regular biases against them due to lack of training data or biased training data used in AI development.
11. Structuring: They help structure the thinking process or the output of the AI in useful or insightful ways. Use of markdown language can aid in the formatting of output structure.
12. Capability Enhancing: A well-designed prompt may be able to push the AI model to enhance its capabilities. When using the prompt, the AI will be able to successfully accomplish tasks or correctly answer questions that it could not answer with a vanilla / simple prompt (for example by just saying “please find the answer to this question”), or could only answer with a significantly lower success rate. The prompt “take a deep breath and think through this problem step by step” is an example of a prompt that helped some AI models improve their math problem solving success rates.

Building with Generative AI

Another important element of my course is the emphasis on moving beyond the chat interface. Chatting with an AI has become the default way we interact with generative AI but students should know that there are many more powerful ways to use this technology through the Application Programming Interface (API) and in particular by combining it with no-code software development tools. By using the API or custom bot-builders (such as custom GPTs) you can move beyond single conversations to building re-usable products, tools and automations. For example, I have built a pretty nifty networking email bot just using the ChatGPT API with Gmail, Zapier.com and Tally.so, which you can try out here:

Networking with Mohammad Keyhani

A business school professor explores how technology is changing entrepreneurship and works on building a portfolio of side projects in public.

DigitVibeMohammad Keyhani

In a later post, I will highlight some of the student projects including many useful custom GPTs, children's storybooks, videos, comic books, software tools, browser plugins, and other interesting things students have done for the relatively open-ended final group project in this course.

Guest Speakers and Student Support

I benefited from brilliant guest speakers who attended the class. Although we will likely bring new guests to the class in 2024, I owe a great debt and gratitude to the guests who enriched our class in 2023:

Leslie Salgado led two sessions on the ethics of generative AI and responsible AI use for our students, and co-authored the article about collective discovery with me. Eric Gonzalez helped our class learn how expert knowledge about photography and videography can be used in prompting image-generating AI tools like Midjourney. He had used AI to imagine what a new event centre in our city could look like. The acclaimed writer Aritha van Herk came to our class to really showcase the creativity that goes into expert human writing and how AI still has a lot of catching up to do. Logan Grasby gave us a lesson on how to design autonomous agents with generative AI, and Clark Lai explained how Motiv studio is already implementing autonomous AI agents and multi-agent frameworks in their projects. Sigmund Holtz gave us a fascinating demo of AI tools being developed by NeoWorlder. Dr. Sarah Elaine Eaton joined our class to discuss generative AI and academic integrity, and that famous question "is it cheating to use AI?"

The course also benefited from support of multiple PhD students who work with me at the Haskayne School of Business. Zahra Jamshidi showcased how the ChatGPT code interpreter or Advanced Data Analysis mode and Wolfram Mathematica's AI notebooks can be used for data analytics. Andishe Ashjari showcased AI tools that can help in the research process. Mick Elliott led the group formation session and helped us better understand Truth and Reconciliation day. Finally, Mahdieh Sarbazvatan supported the course as a teaching assistant. Undergraduate student Arian Haghighat also helped develop content for this course, as did Mitacs Globalink research intern Jhonathan Steven Pulido Flórez who helped put together resources on prompting and character consistency for AI image generators. Tully Huculak assisted in further aspects of the course including graphic design for visual assets and presentation slides.

Sample Lecture and Course Outline

You can get a further glimpse into my course content on generative AI through this sample lecture video published by the University:

If you would like a copy of the course outline please reach out at mkeyhani@ucalgary.ca