21 examples
Flaky test
Test produces inconsistent, unpredictable results.
[ FAQ1 ]
What is a flaky test?
A flaky test is an automated software test that yields inconsistent results—passing or failing intermittently despite no modifications to the code or test environment. Such tests undermine reliability and confidence in automated testing suites, complicating continuous integration and deployment (CI/CD) processes. Common causes include race conditions, timing issues, external dependencies, or non-deterministic logic within the tests themselves. Persistent flaky tests require additional manual effort, delay development cycles, and obscure genuine issues.
[ FAQ2 ]
How to fix flaky tests
To fix flaky tests, first identify the root causes of intermittent failures by carefully reviewing test logs, conditions, and dependencies. Refactor tests to eliminate sources of non-determinism, such as timing dependencies or asynchronous behaviors, by using explicit waits or mocks for external services. Ensure tests are isolated and repeatable, removing dependencies on shared resources or global state. Implement retries cautiously, focusing instead on resolving the underlying instability. Regularly monitor and analyze test suites, immediately addressing flaky behavior to maintain test reliability and accuracy in CI/CD workflows.
diff block
// Before starting the monitoring, we should know nothing about the clock
assert!(client.get_last_known_update_slot(&sysvar_clock).is_none());
// Start the monitoring
- assert!(client.ensure_account_monitoring(&sysvar_clock).is_ok());
+ assert!(client
+ .ensure_account_monitoring(&sysvar_clock)
+ .await
+ .is_ok());
// Wait for a few slots to happen on-chain
- sleep(Duration::from_millis(2000)).await;
+ sleep(Duration::from_millis(2_000)).await;
greptile
style: Consider parameterizing sleep duration to avoid flaky tests on slower networks.
diff block
+import { test, expect } from '@playwright/test';
+
+test('Invite User', async ({ page }) => {
+ await page.goto('http://localhost:3000/app/home');
+ await page
+ .locator('div')
+ .filter({ hasText: /^Invite people$/ })
+ .first()
+ .click();
+ await page.getByRole('textbox', { name: 'buster@bluthbananas.com,' }).click();
+ await page
+ .getByRole('textbox', { name: 'buster@bluthbananas.com,' })
+ .fill('nate+integration-test@buser.so');
+ await page.getByRole('button', { name: 'Send invites' }).click();
+ await page.waitForLoadState('networkidle');
+ await expect(page.getByText('Invites sent').first()).toBeVisible({ timeout: 3000 });
+
+ await page.getByRole('button').filter({ hasText: /^$/ }).first().click();
+ await page.getByRole('link', { name: 'Users' }).click();
+ await expect(page.getByRole('link', { name: 'nate+integration-test@buser.' })).toBeVisible();
+ await expect(page.getByRole('main')).toMatchAriaSnapshot(`
+ - img
+ - text: nate+integration-test@buser.so Restricted Querier
+ `);
+
+ await page.getByRole('link', { name: 'nate+integration-test@buser.' }).click();
+
+ await expect(page.getByText('nate+integration-test@buser.so')).toBeVisible();
+});
+
+test('Can change user role', async ({ page }) => {
+ await page.goto('http://localhost:3000/app/settings/users');
+ await page.getByRole('link', { name: 'B blake blake@buster.so' }).click();
+ await expect(page.getByText('blake@buster.so')).toBeVisible();
+ await expect(page.getByRole('combobox')).toHaveText(/Querier/);
+ await page.getByRole('combobox').click();
+ await page.getByRole('option', { name: 'Workspace Admin' }).click();
+ await expect(
+ page.locator('.text-text-secondary > div:nth-child(2) > .text-text-secondary').first()
+ ).toBeVisible();
+ await page.waitForTimeout(25);
+ await page.waitForLoadState('networkidle');
greptile
logic: Arbitrary 25ms timeout is too short and may cause flaky tests. Either remove or increase to a reasonable value (500ms+)
diff block
+import { expect, test } from '../../utils/playwright-test-base'
+import { mockFeatureFlags } from '../../utils/mockApi'
+import { AppContext } from '~/types'
+
+const VALID_PASSWORD = 'hedgE-hog-123%'
+
+test.describe('Signup', () => {
+ test.beforeEach(async ({ page }) => {
+ await page.click('[data-attr=menu-item-me]')
+ await page.click('[data-attr=top-menu-item-logout]')
+ await expect(page).toHaveURL(/\/login/)
+ await page.goto('/signup')
+ })
greptile
logic: beforeEach assumes user is already logged in before logging out. This may cause flaky tests if initial state varies.
diff block
+from datetime import datetime
+from datetime import timezone
+from unittest.mock import MagicMock
+from unittest.mock import patch
+
+import pytest
+from github import ContentFile
+from github import GithubException
+from github import Repository
+
+from onyx.configs.constants import DocumentSource
+from onyx.connectors.exceptions import ConnectorValidationError
+from onyx.connectors.exceptions import CredentialExpiredError
+from onyx.connectors.exceptions import InsufficientPermissionsError
+from onyx.connectors.github_pages_connector import GitHubPagesConnector
+from onyx.connectors.models import Document
+from onyx.connectors.models import Section
+
+
+@pytest.fixture
+def github_pages_connector() -> GitHubPagesConnector:
+ """Fixture to create a GitHubPagesConnector instance for testing."""
+ connector = GitHubPagesConnector(
+ repo_owner="test_owner",
+ repo_name="test_repo",
+ )
+ connector.load_credentials(
+ {
+ "github_access_token": "test_token",
+ }
+ )
+ return connector
+
+
+def test_github_pages_connector_basic(
+ github_pages_connector: GitHubPagesConnector,
+) -> None:
+ """Test basic functionality of the connector."""
+ with patch.object(github_pages_connector, "_fetch_documents") as mock_fetch:
+ mock_fetch.return_value = [
+ [
+ Document(
+ id="https://test_owner.github.io/test_repo/file1.md",
+ sections=[
+ Section(
+ link="https://test_owner.github.io/test_repo/file1.md",
+ text="Test Content",
+ )
+ ],
+ source=DocumentSource.GITHUB_PAGES,
+ semantic_identifier="file1.md",
+ doc_updated_at=datetime.now(timezone.utc),
+ metadata={
greptile
style: using datetime.now() in tests can lead to flaky tests - consider using a fixed datetime instead
suggested fix
+ doc_updated_at=datetime(2025, 1, 1, tzinfo=timezone.utc),
metadata={
diff block
+import { expect, test } from './utils/posthog-playwright-test-base'
+import { start } from './utils/setup'
+import { pollUntilEventCaptured } from './utils/event-capture-utils'
+
+const startOptions = {
+ options: {
+ capture_performance: {
+ web_vitals: true,
+ },
+ },
+ decideResponseOverrides: {
+ capturePerformance: true,
+ },
+ url: './playground/cypress/index.html',
+}
+
+test.describe('Web Vitals', () => {
+ test('captures web vitals events when enabled', async ({ page, context }) => {
+ await start(startOptions, page, context)
+
+ // Wait for web vitals events to be captured
+ await pollUntilEventCaptured(page, '$web_vitals')
+
+ const webVitalsEvents = (await page.capturedEvents()).filter((event) => event.event === '$web_vitals')
+ expect(webVitalsEvents.length).toBeGreaterThan(0)
+
+ const webVitalsEvent = webVitalsEvents[0]
+ // will always get the FCP event on this bot browser but not necessarily the others
+ expect(webVitalsEvent.properties).toMatchObject({
+ $current_url: expect.any(String),
+ $session_id: expect.any(String),
+ $window_id: expect.any(String),
+ $web_vitals_FCP_value: expect.any(Number),
+ })
+
+ expect(webVitalsEvent.properties.$web_vitals_FCP_event).toMatchObject({
+ name: 'FCP',
+ value: expect.any(Number),
+ $current_url: expect.any(String),
+ $session_id: expect.any(String),
+ $window_id: expect.any(String),
+ timestamp: expect.any(Number),
+ })
+ })
+
+ test('does not capture web vitals when disabled', async ({ page, context }) => {
+ await start(
+ {
+ ...startOptions,
+ options: {
+ capture_performance: {
+ web_vitals: false,
+ },
+ },
+ },
+ page,
+ context
+ )
+
+ // Wait a bit to ensure no web vitals events are captured
+ await page.waitForTimeout(5000)
greptile
style: Using a fixed timeout can lead to flaky tests. Consider using pollUntilEventCaptured with a timeout instead.
diff block
});
+ test("No Histogram should be displayed if Data is not available", async ({ page }) => {
+
+ await logsPage.navigateToLogs();
+ await logsPage.selectIndexStreamDefault();
+ await logsPage.enableSQLMode();
+ await page.locator('[data-test="logs-search-bar-query-editor"]').getByLabel('Editor content;Press Alt+F1').fill('');
+ await page.waitForTimeout(1000);
+ await page.locator('[data-test="logs-search-bar-query-editor"]').getByLabel('Editor content;Press Alt+F1').fill('SELECT count(_timestamp) FROM "default" where code = 201');
+ await page.waitForTimeout(1000);
greptile
style: Arbitrary wait times can lead to flaky tests. Consider using waitForSelector or waitForResponse instead
diff block
+import { randomUUID } from 'crypto';
+
+import { GQL_GENERIC_FIELDS } from 'test/integration/constants/gql-generic-fields.contants';
+import { PERSON_GQL_FIELDS } from 'test/integration/constants/person-gql-fields.constants';
+import { createManyOperationFactory } from 'test/integration/graphql/utils/create-many-operation-factory.util';
+import { createOneOperationFactory } from 'test/integration/graphql/utils/create-one-operation-factory.util';
+import { destroyManyOperationFactory } from 'test/integration/graphql/utils/destroy-many-operation-factory.util';
+import { destroyOneOperationFactory } from 'test/integration/graphql/utils/destroy-one-operation-factory.util';
+import { globalSearchFactory } from 'test/integration/graphql/utils/global-search-factory.util';
+import { makeGraphqlAPIRequest } from 'test/integration/graphql/utils/make-graphql-api-request.util';
+import {
+ createListingCustomObject,
+ LISTING_NAME_PLURAL,
+ LISTING_NAME_SINGULAR,
+} from 'test/integration/metadata/suites/object-metadata/utils/create-test-object-metadata.util';
+import { deleteOneObjectMetadataItem } from 'test/integration/metadata/suites/object-metadata/utils/delete-one-object-metadata.util';
+
+import { ObjectRecord } from 'src/engine/api/graphql/workspace-query-builder/interfaces/object-record.interface';
+
+import { GlobalSearchRecordDTO } from 'src/engine/core-modules/global-search/dtos/global-search-record-dto';
+
+describe('GlobalSearchResolver', () => {
+ let listingObjectMetadataId: { objectMetadataId: string };
+ let people: ObjectRecord[];
+ let listings: ObjectRecord[];
+ let apiKey: ObjectRecord;
+
+ const getSearchRecord = (
+ globalSearch: GlobalSearchRecordDTO[],
+ recordId: string,
+ ) => {
+ return globalSearch.find(
+ (item: GlobalSearchRecordDTO) => item.recordId === recordId,
+ );
+ };
+
+ beforeAll(async () => {
+ listingObjectMetadataId = await createListingCustomObject();
+ listings = (
+ await makeGraphqlAPIRequest(
+ createManyOperationFactory({
+ objectMetadataSingularName: LISTING_NAME_SINGULAR,
+ objectMetadataPluralName: LISTING_NAME_PLURAL,
+ gqlFields: GQL_GENERIC_FIELDS,
+ data: [
+ {
+ id: randomUUID(),
+ name: 'searchInput1',
+ },
+ {
+ id: randomUUID(),
+ name: 'searchInput2',
+ },
+ ],
+ }),
+ )
+ ).body.data.createListings;
+
+ people = (
+ await makeGraphqlAPIRequest(
+ createManyOperationFactory({
+ objectMetadataSingularName: 'person',
+ objectMetadataPluralName: 'people',
+ gqlFields: PERSON_GQL_FIELDS,
+ data: [
+ {
+ id: randomUUID(),
+ name: {
+ firstName: 'searchInput1',
+ lastName: '',
+ },
+ },
+ {
+ id: randomUUID(),
+ name: {
+ firstName: 'searchInput2',
+ lastName: '',
+ },
+ },
+ {
+ id: randomUUID(),
+ name: {
+ firstName: 'searchInput3',
+ lastName: '',
+ },
+ },
+ ],
+ }),
+ )
+ ).body.data.createPeople;
+
+ apiKey = (
+ await makeGraphqlAPIRequest(
+ createOneOperationFactory({
+ objectMetadataSingularName: 'apiKey',
+ gqlFields: GQL_GENERIC_FIELDS,
+ data: {
+ id: randomUUID(),
+ name: 'record not searchable',
+ expiresAt: new Date(Date.now()),
greptile
style: Using Date.now() in tests can lead to flaky tests. Consider using a fixed date for deterministic results.
suggested fix
+ expiresAt: new Date('2025-01-01T00:00:00Z'),
diff block
+import { expect, test, beforeEach } from "bun:test";
+import {
+ fetchLogin,
+ fetchPrivateKey,
+ fetchPrivateKeyBytes,
+ fetchPublicKey,
+ signMessage,
+ encryptData,
+ decryptData,
+ KeyOptions
+} from "../../api";
+import { schnorr } from "@noble/curves/secp256k1";
+
+const TEST_EMAIL = process.env.VITE_TEST_EMAIL;
+const TEST_PASSWORD = process.env.VITE_TEST_PASSWORD;
+const TEST_CLIENT_ID = process.env.VITE_TEST_CLIENT_ID;
+
+if (!TEST_EMAIL || !TEST_PASSWORD || !TEST_CLIENT_ID) {
+ throw new Error("Test credentials must be set in .env.local");
+}
+
+async function setupTestUser() {
+ try {
+ // Try to login
+ const { access_token } = await fetchLogin(TEST_EMAIL!, TEST_PASSWORD!, TEST_CLIENT_ID!);
+ window.localStorage.setItem("access_token", access_token);
+
+ // Add a small delay to ensure tokens are properly set
+ await new Promise((resolve) => setTimeout(resolve, 100));
greptile
style: Consider using a more reliable method than setTimeout for ensuring token setup. Could lead to flaky tests.
diff block
+import { createServer } from 'http'
+import { AddressInfo } from 'net'
+
+import { defaultConfig } from '../../config/config'
+import {
+ HogFunctionInvocation,
+ HogFunctionQueueParametersFetchRequest,
+ HogFunctionQueueParametersFetchResponse,
+} from '../types'
+import { FetchExecutorService } from './fetch-executor.service'
+
+jest.unmock('node-fetch')
+
+describe('FetchExecutorService', () => {
+ jest.setTimeout(1000)
greptile
style: 1000ms timeout may be too aggressive for network tests, consider increasing to avoid flaky tests
diff block
+import pytest
+from typing import cast
+from datetime import datetime
+from braintrust import EvalCase, Score
+from autoevals.llm import LLMClassifier
+from autoevals.boolean import Boolean
+
+from ee.hogai.graph import AssistantGraph
+from ee.hogai.utils.types import AssistantNodeName, AssistantState
+from posthog.schema import HumanMessage, VisualizationMessage
+
+from .conftest import MaxEval
+
+
+class FunnelGeneratorCorrectness(LLMClassifier):
+ """Evaluate funnel generator output correctness."""
+
+ def __init__(self, **kwargs):
+ super().__init__(
+ name="funnel_generator_correctness",
+ prompt_template="""Evaluate if the generated funnel query correctly implements the plan and would answer the user's question effectively.
+
+User question: {{input}}
+Funnel plan: {{context}}
+Generated funnel query: {{output}}
+
+Evaluation criteria:
+1. The funnel steps should match those specified in the plan
+2. Property filters should be implemented correctly, with appropriate operators (contains instead of equals for text)
+3. Exclusion steps should be implemented as specified in the plan
+4. Breakdowns should be implemented as specified in the plan
+5. Time periods should match what's specified in the plan
+
+How would you rate the correctness of the generated funnel query? Choose one:
+- perfect: The query fully implements the plan and would correctly answer the user's question.
+- good: The query implements most of the plan with minor differences that don't affect correctness.
+- partial: The query implements some aspects of the plan but has issues that might affect results.
+- incorrect: The query fails to implement the plan or would not correctly answer the question.""",
+ choice_scores={
+ "perfect": 1.0,
+ "good": 0.7,
+ "partial": 0.4,
+ "incorrect": 0.0,
+ },
+ model="gpt-4.1",
+ **kwargs,
+ )
+
+
+class ContainsInsteadOfEquals(Boolean):
+ """Check if the generator replaced 'equals' with 'contains' for text searches in funnel."""
+
+ def _run_evaluation(self, output, **kwargs):
+ if not output:
+ return Score(name=self._name(), score=0.0)
+
+ json_output = output.model_dump_json(exclude_none=True)
+ if "exact" in json_output:
+ return Score(name=self._name(), score=0.0, comment="Used 'exact' instead of 'contains'")
+
+ # Check if 'icontains' is used for text properties
+ if "icontains" in json_output:
+ # For case-insensitive searching, check if the value is lowercase
+ is_lowercase = False
+ if kwargs.get("expected"):
+ search_term = kwargs["expected"]
+ if search_term.lower() in json_output and search_term not in json_output:
+ is_lowercase = True
+
+ return Score(
+ name=self._name(),
+ score=1.0 if is_lowercase else 0.5,
+ comment="Uses 'icontains' but case handling is " + ("correct" if is_lowercase else "incorrect"),
+ )
+
+ return Score(name=self._name(), score=0.5, comment="No text search operators found")
+
+
+class DateRangeCorrectness(Boolean):
+ """Check if the generator sets the correct date range."""
+
+ def _run_evaluation(self, output, expected, **kwargs):
+ if not output:
+ return Score(name=self._name(), score=0.0)
+
+ if not hasattr(output, "dateRange"):
+ return Score(name=self._name(), score=0.0, comment="No date range specified")
+
+ date_range = output.dateRange
+ if not date_range:
+ return Score(name=self._name(), score=0.0, comment="Empty date range")
+
+ # For tests that check for current year inclusion
+ if expected == "current_year":
+ year = str(datetime.now().year)
+ has_current_year = (date_range.date_from and year in date_range.date_from) or (
+ date_range.date_to and year in date_range.date_to
+ )
greptile
logic: Using datetime.now() in tests can lead to flaky tests when run near year boundaries. Consider injecting the current date or using a fixed reference date.
diff block
+import { test, expect } from '@playwright/test';
+
+test('X axis config - Title', async ({ page }) => {
+ await page.goto('http://localhost:3000/app/home');
+ await page.getByRole('link', { name: 'Metrics', exact: true }).click();
+
+ await page
+ .getByRole('link', {
+ name: 'Yearly Sales Revenue - Signature Cycles Products (Last 3 Years + YTD)'
+ })
+ .click();
+
+ await expect(page.getByTestId('metric-view-chart-content')).toBeVisible();
+ await expect(page.getByTestId('metric-view-chart-content').getByRole('img')).toBeVisible();
+
+ //#1 TEST WE CAN EDIT THE TITLE
+ await page.getByTestId('edit-chart-button').getByRole('button').click();
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ await page.getByRole('textbox', { name: 'Year' }).click();
+ await page.getByRole('textbox', { name: 'Year' }).fill('WOOHOO!');
+ await expect(page.getByTestId('select-axis-drop-zone-xAxis')).toContainText('WOOHOO!');
+ await page.getByRole('button', { name: 'Save' }).click();
+ await page.waitForLoadState('networkidle');
+
+ await page.reload();
+ await expect(page.getByTestId('select-axis-drop-zone-xAxis')).toContainText('WOOHOO!');
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ await page.getByRole('textbox', { name: 'WOOHOO!' }).click();
+ await page.getByRole('textbox', { name: 'WOOHOO!' }).fill('Year');
+ await page.waitForTimeout(100);
+ await page.getByRole('button', { name: 'Save' }).click();
+ await page.waitForLoadState('networkidle');
+ await page.waitForTimeout(100);
+ await page.reload();
+ await expect(page.getByTestId('select-axis-drop-zone-xAxis')).not.toContainText('WOOHOO!');
+});
+
+test('X axis config - we can edit the label style', async ({ page }) => {
+ await page.goto(
+ 'http://localhost:3000/app/metrics/45c17750-2b61-5683-ba8d-ff6c6fefacee/chart?secondary_view=chart-edit'
+ );
+
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ await page.getByTestId('segmented-trigger-percent').click();
+ await expect(page.getByText('Unsaved changes')).toBeVisible();
+ await page.getByRole('button', { name: 'Save' }).click();
+ await page.waitForTimeout(100);
+ await page.waitForLoadState('networkidle');
+ await page.reload();
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ await page.getByTestId('segmented-trigger-number').click();
+ await expect(page.getByText('Unsaved changes')).toBeVisible();
+ await page.getByRole('button', { name: 'Save' }).click();
+ await page.waitForTimeout(100);
+ await page.waitForLoadState('networkidle');
+ await page.reload();
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ expect(page.getByTestId('segmented-trigger-number')).toHaveAttribute('data-state', 'active');
+});
+
+test('X axis config - We can edit the label separator style', async ({ page }) => {
+ await page.goto(
+ 'http://localhost:3000/app/metrics/45c17750-2b61-5683-ba8d-ff6c6fefacee/chart?secondary_view=chart-edit'
+ );
+ await page.getByTestId('select-axis-drop-zone-xAxis').getByRole('button').nth(3).click();
+ await page.getByRole('combobox').click();
+ await page.getByRole('option', { name: '100,000' }).click();
+ await page.getByRole('button', { name: 'Save' }).click();
+
+ await page.waitForTimeout(10);
+ await page.waitForLoadState('networkidle');
greptile
logic: 10ms timeout is too short and may cause flaky tests. Should use a longer timeout or better wait strategy.
diff block
+import { BusterRoutes, createBusterRoute } from '@/routes';
+import { test, expect } from '@playwright/test';
+
+// Define routes
+const homePage = createBusterRoute({
+ route: BusterRoutes.APP_HOME
+});
+const loginPage = createBusterRoute({
+ route: BusterRoutes.AUTH_LOGIN
+});
+
+test.describe('Authentication Flow', () => {
+ test('should redirect when cookies are cleared', async ({ page, context }) => {
+ // First visit home page
+ await page.goto(homePage);
+ await expect(page).toHaveURL(homePage);
+
+ // Clear cookies to remove authentication
+ await context.clearCookies();
+
+ // Try to access the protected home page again
+ await page.goto(homePage);
+ await page.waitForTimeout(25);
+
+ // Should be redirected away from the protected route
+ await expect(page).not.toHaveURL(homePage);
+ await expect(page).toHaveURL(loginPage);
+ });
+
+ test('go to home page', async ({ page }) => {
+ await page.goto(homePage);
+
+ //for 100 milliseconds
+ await page.waitForTimeout(10);
greptile
logic: 10ms timeout is too short and may cause flaky tests - increase to at least 1000ms
diff block
+const realDnsLookup = jest.requireActual('dns/promises').lookup
+jest.mock('dns/promises', () => ({
+ lookup: jest.fn((hostname: string, options?: any) => {
+ return realDnsLookup(hostname, options)
+ }),
+}))
+
+import dns from 'dns/promises'
+import { range } from 'lodash'
+
+import { raiseIfUserProvidedUrlUnsafe, secureRequest, SecureRequestError } from './request'
+
+describe('secureRequest', () => {
+ beforeEach(() => {
+ jest.setTimeout(1000)
+ jest.mocked(dns.lookup).mockImplementation(realDnsLookup)
+ })
+ describe('raiseIfUserProvidedUrlUnsafe', () => {
+ it.each([
+ 'https://google.com?q=20', // Safe
+ 'https://posthog.com', // Safe
+ 'https://posthog.com/foo/bar', // Safe, with path
+ 'https://posthog.com:443', // Safe, good port
+ 'https://1.1.1.1', // Safe, public IP
+ ])('should allow safe URLs: %s', async (url) => {
+ await expect(raiseIfUserProvidedUrlUnsafe(url)).resolves.not.toThrow()
+ })
+
+ it.each([
+ ['', 'Invalid URL'],
+ ['@@@', 'Invalid URL'],
+ ['posthog.com', 'Invalid URL'],
+ ['ftp://posthog.com', 'Scheme must be either HTTP or HTTPS'],
+ ['http://localhost', 'Internal hostname'],
+ ['http://192.168.0.5', 'Internal hostname'],
+ ['http://0.0.0.0', 'Internal hostname'],
+ ['http://10.0.0.24', 'Internal hostname'],
+ ['http://172.20.0.21', 'Internal hostname'],
+ ['http://fgtggggzzggggfd.com', 'Invalid hostname'],
+ ])('should raise against unsafe URLs: %s', async (url, error) => {
+ await expect(raiseIfUserProvidedUrlUnsafe(url)).rejects.toThrow(new SecureRequestError(error))
+ })
+ })
+
+ describe('trackedFetch', () => {
+ // By default security features are only enabled in production but for tests we want to enable them
+
+ it('should raise if the URL is unsafe', async () => {
+ await expect(secureRequest('http://localhost')).rejects.toMatchInlineSnapshot(
+ `[SecureRequestError: Internal hostname]`
+ )
+ })
+
+ it('should raise if the URL is unknown', async () => {
+ await expect(secureRequest('http://unknown.domain.unknown')).rejects.toMatchInlineSnapshot(
+ `[SecureRequestError: Invalid hostname]`
+ )
+ })
+
+ it('should successfully fetch from safe URLs', async () => {
+ // This will make a real HTTP request
+ const response = await secureRequest('https://example.com')
+ expect(response.status).toBe(200)
greptile
style: Making real HTTP requests in tests can cause flaky tests and slow down test execution. Consider mocking the request to example.com
diff block
+import os
+import time
+
+import pytest
+import requests
+
+from tests.integration.common_utils.managers.chat import ChatSessionManager
+from tests.integration.common_utils.managers.settings import SettingsManager
+from tests.integration.common_utils.test_models import DATestSettings
+from tests.integration.common_utils.test_models import DATestUser
+
+
+@pytest.mark.skipif(
+ os.environ.get("ENABLE_PAID_ENTERPRISE_EDITION_FEATURES", "").lower() != "true",
+ reason="Chat retention tests are enterprise only",
+)
+def test_chat_retention(reset: None, admin_user: DATestUser) -> None:
+ """Test that chat sessions are deleted after the retention period expires."""
+
+ # Set chat retention period to 10 seconds
+ retention_days = 10 / 86400 # 10 seconds in days (10 / 24 / 60 / 60)
greptile
style: Using 10 seconds for testing is risky - the TTL task may run immediately after setting retention, causing flaky tests. Consider adding a buffer time.
diff block
+from datetime import datetime, timedelta
+from typing import Optional
+from zoneinfo import ZoneInfo
+
+from posthog.hogql import ast
+from posthog.hogql.visitor import clear_locations
+from posthog.models.utils import UUIDT
+from posthog.schema import ReplayActiveHoursQuery
+from posthog.session_recordings.queries.replay_active_hours_query_runner import ReplayActiveHoursQueryRunner
+from posthog.session_recordings.queries.test.test_session_replay_events import produce_replay_summary
+from posthog.test.base import APIBaseTest, ClickhouseTestMixin, flush_persons_and_events
+
+
+class TestReplayActiveHoursQueryRunner(ClickhouseTestMixin, APIBaseTest):
+ maxDiff = None
+
+ def _create_runner(self, query: ReplayActiveHoursQuery) -> ReplayActiveHoursQueryRunner:
+ return ReplayActiveHoursQueryRunner(team=self.team, query=query)
+
+ def _create_session_replay_events(self, timestamp: str, session_id: Optional[str] = None) -> None:
+ if session_id is None:
+ session_id = f"test-session-{UUIDT()}"
+
+ produce_replay_summary(
+ session_id=session_id,
+ team_id=self.team.pk,
+ first_timestamp=timestamp,
+ last_timestamp=timestamp,
+ distinct_id="test-user",
+ first_url="https://example.io/home",
+ click_count=2,
+ keypress_count=2,
+ mouse_activity_count=2,
+ )
+ flush_persons_and_events()
+
+ def test_query_structure(self):
+ runner = self._create_runner(ReplayActiveHoursQuery(source=None))
+ query = runner.to_query()
+ query = clear_locations(query)
+
+ # Verify the basic structure of the query
+ assert isinstance(query, ast.SelectQuery)
+ assert query.select is not None
+ assert len(query.select) == 8 # hour_block + 7 days
+ assert query.select_from is not None
+ assert isinstance(query.select_from, ast.JoinExpr)
+ assert query.select_from.table is not None
+ assert isinstance(query.select_from.table, ast.SelectQuery)
+ assert query.group_by is not None
+ assert len(query.group_by) == 1
+ assert query.order_by is not None
+ assert len(query.order_by) == 1
+
+ def test_active_hours_calculation(self):
+ # Create events for today and yesterday
+ today = datetime.now(ZoneInfo("UTC"))
+ yesterday = today - timedelta(days=1)
greptile
style: using datetime.now() in tests can lead to flaky tests if run near day boundaries
suggested fix
+ today = datetime(2023, 1, 15, tzinfo=ZoneInfo("UTC")) # Use fixed date for stable tests
yesterday = today - timedelta(days=1)
diff block
db: dbModule.db,
}
}
+
+export function mockScheduleStatusDb({
+ schedule = [
+ {
+ id: 'schedule-id',
+ workflowId: 'workflow-id',
+ status: 'active',
+ failedCount: 0,
+ lastRanAt: new Date('2024-01-01T00:00:00.000Z'),
+ lastFailedAt: null,
+ nextRunAt: new Date('2024-01-02T00:00:00.000Z'),
+ },
+ ],
+ workflow = [
+ {
+ userId: 'user-id',
+ },
+ ],
+}: {
+ schedule?: any[]
+ workflow?: any[]
+} = {}) {
+ vi.doMock('@/db', () => {
+ let callCount = 0
+
+ const select = vi.fn().mockImplementation(() => ({
+ from: vi.fn().mockImplementation(() => ({
+ where: vi.fn().mockImplementation(() => ({
+ limit: vi.fn().mockImplementation(() => {
+ callCount += 1
+ if (callCount === 1) return schedule
+ if (callCount === 2) return workflow
+ return []
+ }),
greptile
logic: The callCount logic could lead to flaky tests if multiple test cases run in sequence. Consider resetting callCount between tests.
diff block
+using System;
+using System.Collections.Generic;
+using Xunit;
+using Phoenix.MarketData.Core; // Add this for PriceSide enum
+using Phoenix.MarketData.Core.Models;
+using Phoenix.MarketData.Domain.Models;
+using Phoenix.MarketData.Infrastructure.Mapping;
+using Phoenix.MarketData.Infrastructure.Serialization;
+
+namespace Phoenix.MarketData.Infrastructure.Tests.Mapping
+{
+ public class FxSpotPriceDataMapperTests
+ {
+ [Fact]
+ public void ToDto_WithValidDomain_ReturnsMappedDto()
+ {
+ // Arrange
+ var mapper = new FxSpotPriceDataMapper();
+ var domain = new FxSpotPriceData
+ {
+ AssetId = "EUR/USD",
+ AsOfDate = DateOnly.FromDateTime(DateTime.Today),
+ AsOfTime = TimeOnly.FromDateTime(DateTime.Now),
greptile
style: Using DateTime.Now in tests can lead to flaky tests. Consider using a fixed datetime
diff block
+import { expect } from "chai";
+import {
+ anchor,
+ web3,
+ FindComponentPda,
+ FindEntityPda,
+ SerializeArgs,
+} from "../../clients/bolt-sdk/lib";
+import { Direction } from "../framework";
+
+export function ecs(framework) {
+ describe("ECS", () => {
+ it("Add entity 1", async () => {
+ const world = await framework.worldProgram.account.world.fetch(
+ framework.worldPda,
+ );
+ framework.entity1Pda = FindEntityPda({
+ worldId: world.id,
+ entityId: world.entities,
+ });
+ const instruction = await framework.worldProgram.methods
+ .addEntity(null)
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ world: framework.worldPda,
+ entity: framework.entity1Pda,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Add entity 2", async () => {
+ const world = await framework.worldProgram.account.world.fetch(
+ framework.worldPda,
+ );
+ framework.entity2Pda = FindEntityPda({
+ worldId: world.id,
+ entityId: world.entities,
+ });
+ const instruction = await framework.worldProgram.methods
+ .addEntity(null)
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ world: framework.worldPda,
+ entity: framework.entity2Pda,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Add entity 3", async () => {
+ const world = await framework.worldProgram.account.world.fetch(
+ framework.worldPda,
+ );
+ const entity3Pda = FindEntityPda({
+ worldId: world.id,
+ entityId: world.entities,
+ });
+ const instruction = await framework.worldProgram.methods
+ .addEntity(null)
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ world: framework.worldPda,
+ entity: entity3Pda,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Add entity 4 (with seed)", async () => {
+ const world = await framework.worldProgram.account.world.fetch(
+ framework.worldPda,
+ );
+ const seed = Buffer.from("custom-seed");
+ framework.entity4Pda = FindEntityPda({ worldId: world.id, seed });
+ const instruction = await framework.worldProgram.methods
+ .addEntity(seed)
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ world: framework.worldPda,
+ entity: framework.entity4Pda,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Initialize Velocity Component on Entity 1 (with seed)", async () => {
+ const componentId = framework.exampleComponentVelocity.programId;
+ framework.componentVelocityEntity1Pda = FindComponentPda({
+ componentId,
+ entity: framework.entity1Pda,
+ seed: "component-velocity",
+ });
+ const instruction = await framework.worldProgram.methods
+ .initializeComponent()
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ entity: framework.entity1Pda,
+ data: framework.componentVelocityEntity1Pda,
+ componentProgram: componentId,
+ authority: framework.worldProgram.programId,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Initialize Position Component on Entity 1", async () => {
+ const componentId = framework.exampleComponentPosition.programId;
+ framework.componentPositionEntity1Pda = FindComponentPda({
+ componentId,
+ entity: framework.entity1Pda,
+ });
+ const instruction = await framework.worldProgram.methods
+ .initializeComponent()
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ entity: framework.entity1Pda,
+ data: framework.componentPositionEntity1Pda,
+ componentProgram: componentId,
+ authority: framework.worldProgram.programId,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Initialize Position Component on Entity 2", async () => {
+ const componentId = framework.exampleComponentPosition.programId;
+ const componentPda = FindComponentPda({
+ componentId,
+ entity: framework.entity2Pda,
+ });
+ const instruction = await framework.worldProgram.methods
+ .initializeComponent()
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ entity: framework.entity2Pda,
+ data: componentPda,
+ componentProgram: componentId,
+ authority: framework.worldProgram.programId,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Initialize Position Component on Entity 4", async () => {
+ const componentId = framework.exampleComponentPosition.programId;
+ framework.componentPositionEntity4Pda = FindComponentPda({
+ componentId,
+ entity: framework.entity4Pda,
+ });
+ const instruction = await framework.worldProgram.methods
+ .initializeComponent()
+ .accounts({
+ payer: framework.provider.wallet.publicKey,
+ entity: framework.entity4Pda,
+ data: framework.componentPositionEntity4Pda,
+ componentProgram: componentId,
+ authority: framework.worldProgram.programId,
+ })
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+ });
+
+ it("Check Position on Entity 1 is default", async () => {
+ const position =
+ await framework.exampleComponentPosition.account.position.fetch(
+ framework.componentPositionEntity1Pda,
+ );
+ expect(position.x.toNumber()).to.equal(0);
+ expect(position.y.toNumber()).to.equal(0);
+ expect(position.z.toNumber()).to.equal(0);
+ });
+
+ it("Apply Simple Movement System (Up) on Entity 1", async () => {
+ const instruction = await framework.worldProgram.methods
+ .apply(SerializeArgs({ direction: Direction.Up }))
+ .accounts({
+ authority: framework.provider.wallet.publicKey,
+ boltSystem: framework.systemSimpleMovement.programId,
+ world: framework.worldPda,
+ sessionToken: null,
+ })
+ .remainingAccounts([
+ {
+ pubkey: framework.exampleComponentPosition.programId,
+ isSigner: false,
+ isWritable: false,
+ },
+ {
+ pubkey: framework.componentPositionEntity1Pda,
+ isSigner: false,
+ isWritable: true,
+ },
+ ])
+ .instruction();
+
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+
+ const position =
+ await framework.exampleComponentPosition.account.position.fetch(
+ framework.componentPositionEntity1Pda,
+ );
+ expect(position.x.toNumber()).to.equal(0);
+ expect(position.y.toNumber()).to.equal(1);
+ expect(position.z.toNumber()).to.equal(0);
+ });
+
+ it("Apply Simple Movement System (Right) on Entity 1", async () => {
+ const instruction = await framework.worldProgram.methods
+ .apply(SerializeArgs({ direction: Direction.Right }))
+ .accounts({
+ authority: framework.provider.wallet.publicKey,
+ boltSystem: framework.systemSimpleMovement.programId,
+ world: framework.worldPda,
+ sessionToken: null,
+ })
+ .remainingAccounts([
+ {
+ pubkey: framework.exampleComponentPosition.programId,
+ isSigner: false,
+ isWritable: false,
+ },
+ {
+ pubkey: framework.componentPositionEntity1Pda,
+ isSigner: false,
+ isWritable: true,
+ },
+ ])
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+
+ const position =
+ await framework.exampleComponentPosition.account.position.fetch(
+ framework.componentPositionEntity1Pda,
+ );
+ expect(position.x.toNumber()).to.equal(1);
+ expect(position.y.toNumber()).to.equal(1);
+ expect(position.z.toNumber()).to.equal(0);
+ });
+
+ it("Apply Fly System on Entity 1", async () => {
+ const instruction = await framework.worldProgram.methods
+ .apply(SerializeArgs())
+ .accounts({
+ authority: framework.provider.wallet.publicKey,
+ boltSystem: framework.systemFly.programId,
+ world: framework.worldPda,
+ sessionToken: null,
+ })
+ .remainingAccounts([
+ {
+ pubkey: framework.exampleComponentPosition.programId,
+ isSigner: false,
+ isWritable: false,
+ },
+ {
+ pubkey: framework.componentPositionEntity1Pda,
+ isSigner: false,
+ isWritable: true,
+ },
+ ])
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+
+ const position =
+ await framework.exampleComponentPosition.account.position.fetch(
+ framework.componentPositionEntity1Pda,
+ );
+ expect(position.x.toNumber()).to.equal(1);
+ expect(position.y.toNumber()).to.equal(1);
+ expect(position.z.toNumber()).to.equal(1);
+ });
+
+ it("Apply System Velocity on Entity 1", async () => {
+ const instruction = await framework.worldProgram.methods
+ .apply(SerializeArgs())
+ .accounts({
+ authority: framework.provider.wallet.publicKey,
+ boltSystem: framework.systemApplyVelocity.programId,
+ world: framework.worldPda,
+ sessionToken: null,
+ })
+ .remainingAccounts([
+ {
+ pubkey: framework.exampleComponentVelocity.programId,
+ isSigner: false,
+ isWritable: false,
+ },
+ {
+ pubkey: framework.componentVelocityEntity1Pda,
+ isSigner: false,
+ isWritable: true,
+ },
+ {
+ pubkey: framework.exampleComponentPosition.programId,
+ isSigner: false,
+ isWritable: false,
+ },
+ {
+ pubkey: framework.componentPositionEntity1Pda,
+ isSigner: false,
+ isWritable: true,
+ },
+ ])
+ .instruction();
+ const transaction = new anchor.web3.Transaction().add(instruction);
+ await framework.provider.sendAndConfirm(transaction);
+
+ const velocity =
+ await framework.exampleComponentVelocity.account.velocity.fetch(
+ framework.componentVelocityEntity1Pda,
+ );
+ expect(velocity.x.toNumber()).to.equal(10);
+ expect(velocity.y.toNumber()).to.equal(0);
+ expect(velocity.z.toNumber()).to.equal(0);
+ expect(velocity.lastApplied.toNumber()).to.not.equal(0);
+
+ const position =
+ await framework.exampleComponentPosition.account.position.fetch(
+ framework.componentPositionEntity1Pda,
+ );
+ expect(position.x.toNumber()).to.greaterThan(1);
+ expect(position.y.toNumber()).to.equal(1);
+ expect(position.z.toNumber()).to.equal(1);
+ });
+
+ it("Apply System Velocity on Entity 1, with Clock external account", async () => {
greptile
logic: test case uses hardcoded z-value expectation (300) which may be time-dependent and cause flaky tests
diff block
+import datetime
+
+import pytest
+from batchgrader.processing_stats import ProcessingStats
+
+
+def test_duration_with_start_and_end_times() -> None:
+ """
+ If start_time and end_time are set, duration returns the correct difference in seconds.
+ """
+ stats = ProcessingStats()
+ stats.start_time = datetime.datetime(2025, 5, 20, 12, 0, 0)
+ stats.end_time = datetime.datetime(2025, 5, 20, 12, 0, 30)
+ assert stats.duration == 30.0
+
+
+def test_duration_without_end_time() -> None:
+ """
+ If end_time is not set, duration returns None.
+ """
+ stats = ProcessingStats()
+ stats.start_time = datetime.datetime.now()
+ stats.end_time = None
+ assert stats.duration is None
greptile
style: Using datetime.now() in tests can lead to flaky tests. Consider using a fixed datetime value instead.
suggested fix
stats = ProcessingStats()
stats.start_time = datetime.datetime(2025, 5, 20, 12, 0, 0)
stats.end_time = None
assert stats.duration is Nonediff block
+import { Delivery } from "./delivery";
+import { PackageMap } from "./package";
+
+export const debugDeliveries: Delivery[] = [
+ {
+ id: "E2836A4F-8BC9-463D-A703-EDFF8D1580D9",
+ name: "undelivered package that is estimated ahead",
+ trackingNumber: "1Zasdf",
+ carrier: "ups",
+ debug: true,
+ },
+ {
+ id: "E04FEF49-47DD-4D67-A60B-5D9CC6F78C1E",
+ name: "delivery date in the past that was delivered",
+ trackingNumber: "92462346326",
+ carrier: "fedex",
+ debug: true,
+ },
+ {
+ id: "D9A04926-2A37-40C1-B2C5-DB2E1924F0A8",
+ name: "delivery today that is undelivered",
+ trackingNumber: "156845865089045685454467",
+ carrier: "usps",
+ debug: true,
+ },
+ {
+ id: "195A6D9E-81BC-4CEC-97B7-DB5D5A96E8BC",
+ name: "delivery that is unknown",
+ trackingNumber: "934724536784235786435786",
+ carrier: "usps",
+ debug: true,
+ },
+ {
+ id: "13DCF84F-EEE5-4827-AD08-4C0F336E87BB",
+ name: "partially delivered",
+ trackingNumber: "1Zdflgjadlhjfgasdfasdf",
+ carrier: "ups",
+ debug: true,
+ },
+ {
+ id: "CF022134-EEE4-4AA1-BD74-64BF36B465FD",
+ name: "partially delivered, with unknown delivery dates",
+ trackingNumber: "134578458906534",
+ carrier: "fedex",
+ debug: true,
+ },
+ {
+ id: "DD17AC8D-9048-43ED-AF35-4195A2F97243",
+ name: "no packages",
+ trackingNumber: "198451726304587",
+ carrier: "fedex",
+ debug: true,
+ },
+];
+
+export const debugPackages: PackageMap = {};
+debugPackages[debugDeliveries[0].id] = {
+ packages: [
+ {
+ deliveryDate: new Date(new Date().getTime() + 1000 * 60 * 60 * 24 * 4), // 4 days ahead
+ delivered: false,
+ activity: [],
+ },
+ ],
+};
+debugPackages[debugDeliveries[1].id] = {
+ packages: [
+ {
+ deliveryDate: new Date("2024-11-09"),
+ delivered: true,
+ activity: [],
+ },
+ ],
+};
+debugPackages[debugDeliveries[2].id] = {
+ packages: [
+ {
+ deliveryDate: new Date(),
greptile
style: Using new Date() without time component could cause flaky tests if run near midnight. Consider setting a specific time of day.
diff block
+#!/bin/bash
+
+# Integration Tests for Raycast Alarms Trigger Functionality
+# These tests verify the alarm triggering mechanism
+
+# Load test utilities
+source "$(dirname "$(dirname "$0")")/test-utils.sh"
+
+# Test: Simulation of alarm triggering
+test_alarm_trigger() {
+ local test_id="trigger_test_alarm_$(date +%s)"
+ local test_title="Trigger Test Alarm"
+ local test_sound="/System/Library/Sounds/Submarine.aiff"
+
+ # Create an active directory
+ mkdir -p "$TEST_CONFIG_DIR/active"
+
+ # Run the trigger script with nohup to avoid actual sound playing and dialogs
+ # Just test that the script executes without errors
+ nohup "$TEST_CONFIG_DIR/scripts/trigger-alarm.sh" "$test_id" "$test_title" "$test_sound" "0" > "$TEST_CONFIG_DIR/nohup.out" 2>&1 &
+
+ # Store process ID
+ local pid=$!
+
+ # Wait a moment for it to initialize
+ sleep 1
greptile
style: Sleep 1 is not reliable for waiting on file creation - could lead to flaky tests. Consider using a retry loop with timeout instead
Want to avoid this bug in your codebase? Try Greptile.
Avoid this bug!