Monitoring AI Prompts with FortiGate and Fastvue Reporter

Although you can block ChatGPT using your organization's firewall, ChatGPT is built into Microsoft Bing; Notion launched Notion AI, you can install GPT add-ons in Google Sheets, and hundreds of AI Writing assistants are popping up every day such as Rytr, HyperWrite, ParagraphAI, and WordTune.

The AI genie is out of the bottle, and blocking them is like playing a game of whac-a-mole!

Monitoring AI prompts instead

A better approach may be to allow students and staff to use AI Tools such as ChatGPT and Google Bard and monitor the prompts they're entering to make sure it is being used responsibly.

Fortunately, Fortinet FortiGate can now log ChatGPT and Google Bard prompts (tested in firmware v7.2.5 build1517), and at Fastvue we have added a new sections to our reports showing AI Prompts as well as Keyword Matched AI Prompts.

In this article, I'll show you how to set up Fortinet FortiGate's Application Control to log the prompts users are entering into ChatGPT and Google Bard, and then report on those prompts using Fastvue Reporter.

Configuring FortiGate's Application Control to log AI prompts

Fortinet FortiGate comes with a range of Application Signatures that are constantly updated, and recently Fortinet released signatures for ChatGPT Posts, and Google Bard Posts.

When your Application Control sensor has these applications set to Monitor, it will allow the application, and log extra details that include the Prompt text.

⚠️ Please note that Deep Packet Inspection is required for this functionality to work, so make sure your outbound web access policies have a Full SSL Inspection profile applied and you have deployed the CA Certificate used in that inspection profile to all endpoints going through that policy.

To configure FortiGate's Application Control to monitor AI Prompts:

1. In Fortinet FortiGate, go to Security Profiles > Application Control and edit the Application Control sensor that you're using in your outbound web policies. If you have multiple Application Control sensors used for different outbound policies, you'll need to modify each one as below.

2. In the Application and Filter Overrides section, click Create New.

3. Make sure the Action is set to Monitor and then search for ChatGPT.

4. Click Add All Results and you should see a green tick appear next to all the OpenAI.ChatGPT signatures. The one that will log prompts is OpenAI.ChatGPT_Post, but you may as well add them all here.

5. Do another search for Bard.

6. Again, click Add All Results and make sure a green tick is displayed next to all the Google.Bard signatures.

7. Click OK. You should see all the OpenAI.ChatGPT and Google.Bard signatures in the list with the Action set to Monitor.

   

8. Click OK to save the changes to the Application Sensor.

Assuming this Application Sensor is applied to your outbound web access Firewall Policies (Policy & Objects > Firewall Policy > Edit > Application Control), you can now test your FortiGate is logging AI prompts.

Logging AI prompts in Fortinet FortiGate

To test your Fortinet FortiGate is now logging AI prompts from ChatGPT and Google Bard.

⚠️ As AI prompts can be very long, it's important to note that FortiGate will only the first 255 characters of any prompt.

1. Go to Log & Report > Security Events > Logs and select Application Control from the dropdown.

2. In a separate tab or window go to ChatGPT (chat.openai.com), login, and enter a prompt.

3. You should see this prompt appear in the Application Control log. You can also filter the log by Application Name = OpenAI.ChatGPT_Post

   

4. Repeat the same test using Google Bard. You can filter the Application Control log by Application Name = Google.Bard_Post

   

⚠️ Troubleshooting: If you're not seeing these events in the Application Control log, make sure the browser you're conducting the tests with has your FortiGate's SSL certificate applied. To do this in Google Chrome, click the lock in the address bar and click Connection is Secure > Certificate is Valid and make sure this is the certificate you're using in your SSL/SSH Inspection Profile.

Also make sure your machine is going through the policy you have configured with the edited Application Control Sensor above.

Reporting on AI Prompts in Fastvue Reporter

Now that you're logging AI prompts (at least in ChatGPT and Google Bard), you can report on these prompts in Fastvue Reporter.

Before you start, make sure you are running Fastvue Reporter for FortiGate version 1.0.1.74 or above (Settings > About). If not, please download the latest version from our website.

To generate a report showing AI Prompts:

1. In Fastvue Reporter, go to Reports > Overview Report > Internet Usage

2. Select your desired date range (Optional: Click the Filters button to apply any filters such as Security Groups etc).

3. Click Run Report.

4. On the left hand side, go to the Safeguarding > AI Prompts section.

5. This section shows Keyword Matched Prompts by default which may be blank if no prompts have been entered that match our Keyword Groups (See Settings > Keywords). To see all prompts, click the All Prompts button at the top of the AI Prompts widget.

Summary

Many schools have made the decision to block AI tools such as ChatGPT and Google Bard to avoid students using these tools to cheat on school work, however the number of AI tools is becoming quite prolific, and blocking access to them is becoming more difficult.

AI is here to stay, and one approach many schools are now taking is to embrace the benefits of AI tools and teach students how to utilise them responsibly. Of course, that is easier said than done, so monitoring how students utilise AI tools is a critical part of this process.

Fastvue Reporter, together with Fortinet FortiGate's Application Control can help you monitor and report on AI prompts students (and staff) enter into AI tools and send these reports to the right people.

Fastvue Reporter's Keywords Functionality can also detect inappropriate AI prompts related to Self-harm, Extremism, Drugs, or Adult and Profanity, and you can add your own custom keywords such as 'write essay' to help detect students using these tools to cheat.