Python 88 – Guide to Tokenize AI Prompt to Save Cost & Generate Better Result

Generative AI API is not free as well as search engines to provide information resources, although the pricing varies by service providers. Some are not pricey, however, some Generative AI service providers charge more and it’s not cheap, although it also depends on how users optimize their prompt.

In this article, I’ll walk through an approach of tokenized context materials to save cost and deliver better results using any Generative AI API. There are two sections in this article. First is a video to walk through the whole process, and the other section is to elaborate things related to modules, APIs, experimental comparison results between using tokenized context and non-tokenized context.

• Modules, APIs, and Functions Preparation
• Input Character Number Comparison in Prompt
• Generated Content Quality Comparison
• Full Python script of content tokenizer and Generative AI prompt for full piece of article

Modules, APIs, and Functions Preparation

Regarding this approach, we need 3 main modules and 2 functions, which are as follows:

• Content tokenizer
• Generative AI Module
• Requests module

Instead of using a detailed Python script to implement the experiment, we use two API endpoints offered from 3rd party service providers. For more details regarding the Python script, please subscribe to our newsletter with a message about what Python script you are interested in.

Here are the coding samples:

import requests

from tokenAuthenticationn import BuyfromloAuthentication

"""Tokenizer & Blogchain API """

apiURL = "https://www.buyfromlo.com/"

## Buyfrommlo APIs ##

tokenizerEndpoint = 'api/1/word-tokenizer'

blogchainEndpoint = 'api/3/blogchain'

""" Buyfromlo Authentication """

authentication = BuyfromloAuthentication.BFLToken()

gpt4apiKey = BuyfromloAuthentication.gpt4APIkey()

llmversion = BuyfromloAuthentication.llmversionv1()

Furthermore, we need to create two functions which one is used to tokenize the context information, and the other one is used to generate the article.

""" Functions """

def wordTokenizer2024(apiEndpoint, rawContent, token123=authentication):

   url = requests.post(

       apiEndpoint + "?token=" + token123,

       json={

           'originalContent':rawContent

           })

   return url

def blogchain2024(apiEndpoint, originalTokens, topicName, role123, sectionBlog, language123, model123, apiKey123, token123=authentication):

   url = requests.post(

       apiEndpoint + '?token=' + token123,

       json={

           "originalContent": originalTokens,

           "topic_name": topicName,

           "role":role123,

           "tableofcontentquantity": sectionBlog,

           "language":language123,

           "llmModel":model123,

           "apikey":apiKey123})

   return url

Length of Character Number Comparison in Prompt

Less input characters that meanwhile deliver the same result represent less token consumption. Thus, we are able to generate and compare original non-tokenized context with tokenized context using the tokenizer API. Here is the result from English language as follows for reference:

Non-tokenized context: 1543

Tokenized context: 2460

Generated Content Quality Comparison

Less characters doesn’t mean the final content quality is better, after all our purpose is to generate expected and good quality of content using Generative AI instead of just saving dollars. Thus, we need to try using them separately to generate the full article and compare the quality. Below are the result, which before represents using non-tokenized context and after represents using tokenized context.

Before: Result generation with non-tokenized context materials in AI prompt

Headline: Tesla Restricts Autopilot Usage for Enhanced Driver Safety

--------------------

Tesla is recalling over two million vehicles as a result of an ongoing investigation by the top automotive safety regulator. The recall will restrict the use of Autosteer, a headline feature of Tesla's basic Autopilot software. Tesla's driver-assist system still claims to operate dominantly over rival systems. The basic Autopilot comes standard in Tesla vehicles and includes features like traffic-aware cruise control. The National Highway Traffic Safety Administration (NHTSA) released documents stating that Tesla cars will now check to see if the driver is paying attention to the road when using Autosteer. Tesla will soon ship over-the-air software updates to add additional controls and alerts to encourage drivers to stay alert while using Autosteer. Tesla has long said that Autosteer features are intended for use on controlled-access highways with fully attentive drivers. The company has rarely walked back capabilities of its cars, perpetually promoting the idea of a fully self-driving future. Tesla is facing a number of lawsuits involving Autopilot, and one in California has already gone Tesla's way. The California Department of Motor Vehicles accused Tesla of false advertising regarding the capabilities of Autopilot. The NHTSA documents reveal that the agency began meeting with Tesla in October to address driver misuse of Autosteer, and Tesla concurred with the analysis. The ultimate decision on the recall was made in December.

--------------------

Table of Contents:

• Tesla Autopilot: A Game-Changing Driver-Assist Software
• Tesla Autopilot: The Controversy Surrounding its Use and Capabilities

--------------------

Tesla's Autopilot, a groundbreaking driver-assistance software, has revolutionized the automotive industry. This advanced technology allows vehicles to stay centered in their lane, even around large curves, providing invaluable assistance to drivers. However, recent events have highlighted certain limitations of Autopilot, prompting Tesla to issue a recall affecting over two million vehicles. The National Highway Traffic Safety Administration (NHTSA) expressed concerns regarding the system's ability to check if drivers are paying attention to the road while using Autopilot. To address these concerns, Tesla will soon release an over-the-air software update that will add additional control alerts, encouraging drivers to remain alert while using Autopilot. Despite these limitations, Tesla emphasizes that the Autopilot feature is intended for use on controlled-access highways and requires the full attention of the driver.

Tesla's Autopilot system, a driver-assistance feature that enables cars to stay centered in their lane and navigate around curves, has been the subject of much controversy. While Tesla claims that Autopilot is safer than human drivers, critics argue that the system is not yet advanced enough to be used without human supervision. The controversy surrounding Autopilot intensified in 2022 when the National Highway Traffic Safety Administration (NHTSA) launched an investigation into the system following a series of accidents involving Tesla vehicles. The NHTSA is particularly concerned about the fact that drivers may become complacent when using Autopilot and may not be paying enough attention to the road.

After: Used tokenized context materials in AI prompt

Headline: Tesla Restricts Autopilot Software Usage: New Limitations Imposed

--------------------

Tesla is recalling around two million vehicles as a result of a multiyear investigation by the nation's top automotive safety regulator. The recall will restrict the use of Autosteer, a headline feature of Tesla's basic Autopilot software, which allows the car to stay in its lane and navigate curves. Tesla's driver-assist system still claims to operate better than its rivals. The recall also includes additional features such as Traffic-Aware Cruise Control. Tesla will soon ship over-the-air software updates to add additional controls and alerts to encourage driver attentiveness while using Autosteer. 

The company has always emphasized that Autosteer is intended for use on controlled-access highways with a fully attentive driver. Tesla has faced lawsuits and regulatory probes over its Autopilot system, with allegations of false advertising and concerns about misuse by drivers. The National Highway Traffic Safety Administration (NHTSA) began meetings with Tesla in October 2021 to address driver misuse of Autosteer, and both parties ultimately agreed to the recall in December.

--------------------

Table of Contents:

• Tesla Autopilot: The Evolution and Recalls
• Tesla Autopilot: Safety Concerns and Government Investigations

--------------------

Tesla Autopilot, the advanced driver-assistance system developed by Tesla, has undergone significant evolution since its introduction. While it offers impressive features that assist drivers in various driving scenarios, it has also faced its share of challenges and recalls. One notable recall involved over two million vehicles due to concerns about the system's ability to properly identify emergency vehicles parked on the side of the road. 

This issue came to light as a result of an ongoing multi-year investigation by the National Highway Traffic Safety Administration (NHTSA), the top automotive safety regulator in the United States. The recall restricted the use of Autosteer, one of the headline features of Tesla's Autopilot software, which allows the car to stay planted in the center of its lane, even around big curves. Despite these challenges, Tesla maintains that its driver-assist system still operates dominantly compared to rivals. The basic Autopilot software comes standard on all Tesla vehicles and also includes features called Traffic Aware Cruise Control, which operates much like the adaptive cruise control found in modern cars, specifically for highway traffic.

The Tesla Autopilot system has come under increasing scrutiny in recent years due to safety concerns and government investigations. While Tesla claims that Autopilot is the safest driver-assist system on the market, critics argue that it can lull drivers into a false sense of security and lead to accidents. 

In 2022, the National Highway Traffic Safety Administration (NHTSA) opened an investigation into Tesla's Autopilot system after a series of crashes involving the technology. The investigation is ongoing, but the NHTSA has already found that Tesla's Autopilot system is insufficient in ensuring that drivers are paying attention to the road. Tesla has responded by issuing a series of over-the-air software updates that add additional controls and alerts to the Autopilot system. However, some experts believe that these updates are not enough to address the fundamental safety concerns with Autopilot. The company has also been accused of falsely advertising the capabilities of Autopilot, leading to a number of lawsuits.

Full Python script of content tokenizer and Generative AI prompt for full piece of article

If you are interested in Chapter 88 – Guide to Tokenize AI Prompt to Save Cost & Generate Better Result, please subscribe to our newsletter by adding the message ‘Chapter 88 + Python script’. We would send you the script immediately to your mailbox.

I hope you enjoy reading Chapter 88 – Guide to Tokenize AI Prompt to Save Cost & Generate Better Result. If you did, please support us by doing one of the things listed below, because it always helps out our channel.

• Support and Donate to our channel through PayPal (paypal.me/Easy2digital)
• Subscribe to my channel and turn on the notification bell Easy2Digital Youtube channel.
• Follow and like my page Easy2Digital Facebook page
• Share the article on your social network with the hashtag #easy2digital
• You sign up for our weekly newsletter to receive Easy2Digital latest articles, videos, and discount codes
• Subscribe to our monthly membership through Patreon to enjoy exclusive benefits (www.patreon.com/louisludigital)