StreetViewPlayerCamera.cs

using System;
using System.Collections;
using CityGen3D;
using UnityEngine;
using UnityEngine.Networking;

#region MIT LICENSE

/*
    License: The MIT License (MIT)
    Copyright (C) 2020 Shannon Rowe

    Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
    documentation files (the "Software"), to deal in the Software without restriction, including without limitation
    the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
    and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of
    the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
    CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    DEALINGS IN THE SOFTWARE.
*/

#endregion

#region INSTRUCTIONS

/*
    Setup:

    1. Attach this script to an empty game object called Player (or similar) to be the player controller.
    2. Drag or create the main camera as a child under Player, and attach it to "mainCamera" field in the inspector.
       Set its Clear Flags to Solid Color and its Culling Mask to Everything, and reset all transform values to zero.
    3. Create a second camera (remove the redundant audio listener) as a child, also under Player,
       and attach it to "renderCamera" in the inspector for this script.
       Set its Clear Flags to Skybox and its Culling Mask to Nothing, and again reset transform values to zero.
    4. Attach the landscape GameObject created by CityGen3D to the "landscape" field in the inspector.
    5. Add your Google StreetView API key to the relevant field in the inspector.
       To get one, create a free Google Cloud Platform developer account and enable the Street View Static API.
       On the free pricing tier, this should get you around 25,000 free tile fetches a month, however it is
       advisable to regularly keep an eye on your usage using the developer console to make sure you
       don't inadvertently tip over into the paid threshold. Each panorama used here consists of 6 tiles which
       are downloaded and then folded up into a cubemap.

    Controller Usage:

    This script is a pretty standard flying camera controller that uses WASD/arrows to fly, shift to sprint,
    E/PgUp to ascend or Q/PgDn to descend, and the mouse to turn the camera. Speeds are adjustable in the inspector.
    There's a little red pulsing ball that serves as a crosshairs, but you can turn that off if you prefer.
    If you left-click, it will teleport to where the crosshairs are pointing (and also triggers loading the
    StreetView image - more on that below). If you right-click, it will return you to wherever the Player transform
    was when you first entered play mode, which is useful for setting a starting point to return to.
    The player starts in free movement mode, meaning no gravity and the ability to clip through colliders; if you
    press P it will toggle physics on or off (creating the physics components if you don't have any on the player).
    Lastly, Escape will release the cursor so you can get back to the editor, and left-click will re-enter fly mode.

    StreetView Display usage:

    To open the window, look for it under the CityGen3D menu for "StreetView Display".
    It's a basic editor window that starts out floating but can be docked or resized as per normal window behaviour.
    The resizing defaults to keeping the image square to match the render aspect ratio, but you can change the
    DefaultKeepImageSquare constant in StreetViewWindow if you prefer it to be unconstrained (albeit distorted).

    The magic happens when you enter play mode and fly around and then teleport somewhere with left-click.
    The Google API will be polled and (assuming you have a StreetView Display window open) the panorama will
    appear shortly afterwards in the window, and should rotate in sync with your camera movements.  Note that the
    view is set to attempt to match the map's latitude/longitude position to the equivalent Unity position
    corresponding to roughly where you clicked to teleport, so once you start moving around again, the illusion will
    quickly be lost.  But while you stay in place, you can rotate around and enjoy the synchronised view.

    Technical notes:

    This tool uses the Unity scene skybox to stash the panorama and allow it to be panned around.  This will
    therefore replace any skybox you normally have in your scene.  There may be some adjustments that can be made
    to avoid this, but I haven't explored any of that as this tool isn't intended for use outside the editor anyway.
    You will also notice seams in many of the panoramas - this occurs due to the fairly crude method used for
    stitching the tiles together, which would take a better mathematician than myself to resolve.
    Similarly, I haven't bothered with any of the zoom levels for the tiles that are available in the API, as the
    results are good enough and Google have resolution limits for the free tier anyway, so while if you resize
    the window the size of the image will increase, the level of detail will not.
    Finally, when you left-click somewhere to teleport, you might notice that the player is not always placed
    exactly at the point you clicked.  All Google StreetView panoramas cover a bounding box, so when you click,
    the API call will first get the nearest panorama to where you clicked, and then take the location stored in
    the panorama metadata to derive a revised Unity position that approximates the metadata location, and teleports
    the player controller to that point rather than the point you clicked.  In practice, these are usually so
    close together as to be unnoticeable, however in some sparsely covered regions, you might see larger jumps.
    The reason for this repositioning is so that your view will more closely be in sync between Unity and StreetView.
*/

#endregion

public class StreetViewPlayerCamera : MonoBehaviour
{
    // This is the approximate height of a Google StreetView car camera (8.2 feet)
    private const float CameraHeight = 2.49936f;
    private const PrimitiveType MarkerPrimitive = PrimitiveType.Sphere;
    private const string MarkerObjectName = "Marker";
    private const float MarkerScaleSpeed = 10f;
    private const float MarkerScaleAmount = 0.015f;
    private const float MarkerRotationSpeed = 100f;
    private const float MarkerPositionOffset = 0.25f;
    private const int NumberSides = 6;
    private const int TileWidth = 400;
    private const int TileHeight = 400;
    private const int Fov = 90;
    private const float WaitTimeout = 10f;
    private const string DoublePrecision = "F6";
    private const float DefaultCapsuleColliderHeight = 3f;
    private const float DefaultCapsuleColliderRadius = 2f;
    private const CollisionDetectionMode DefaultCollisionDetectionMode = CollisionDetectionMode.Discrete;

    [SerializeField] private string streetViewApiKey;
    [SerializeField] private Camera mainCamera;
    [SerializeField] private Camera renderCamera;
    [SerializeField] private Landscape landscape;
    [SerializeField] private float walkSpeed = 100f;
    [SerializeField] private float runSpeed = 250f;
    [SerializeField] private float rotationSpeed = 4f;
    [SerializeField] private bool invertYAxis;
    [SerializeField] private bool showMarker = true;
    [SerializeField] private Color markerColor = Color.red;

    public delegate void ImageLoadedDelegate (LoadChain chain);

    public static ImageLoadedDelegate onImageLoaded;

    private Transform _trans;
    private Transform _mainCameraTrans;
    private Transform _renderCameraTrans;
    private Transform _markerTrans;
    private Collider _collider;
    private Rigidbody _rigidbody;
    private Vector2 _currentRotation;
    private Vector2 _restartRotation;
    private Vector3 _restartPosition;
    private GameObject _marker;
    private readonly int _frontTex = Shader.PropertyToID ("_FrontTex");
    private readonly int _leftTex = Shader.PropertyToID ("_LeftTex");
    private readonly int _backTex = Shader.PropertyToID ("_BackTex");
    private readonly int _rightTex = Shader.PropertyToID ("_RightTex");
    private readonly int _upTex = Shader.PropertyToID ("_UpTex");
    private readonly int _downTex = Shader.PropertyToID ("_DownTex");
    private bool _isWorkingOnMetadata;
    private bool _isWorkingOnPanorama;
    private int _fetchesDone;
    private float _timeoutTimer;
    private LoadChain _loadChain;
    private Action _metadataCallbackAction;
    private Action _imageCallbackAction;
    private readonly object _lockObject = new object ();

    private void Awake ()
    {
        _trans = transform;
        _mainCameraTrans = mainCamera.transform;
        _renderCameraTrans = renderCamera.transform;
        _collider = GetComponent<Collider> ();
        if (_collider == null)
        {
            _collider = gameObject.AddComponent<CapsuleCollider> ();
            ((CapsuleCollider) _collider).height = DefaultCapsuleColliderHeight;
            ((CapsuleCollider) _collider).radius = DefaultCapsuleColliderRadius;
        }

        _rigidbody = GetComponent<Rigidbody> ();
        if (_rigidbody == null)
        {
            _rigidbody = gameObject.AddComponent<Rigidbody> ();
            _rigidbody.freezeRotation = true;
            _rigidbody.collisionDetectionMode = DefaultCollisionDetectionMode;
        }

        _collider.enabled = false;
        _rigidbody.isKinematic = true;

        _currentRotation = _mainCameraTrans.eulerAngles / rotationSpeed;
        _restartRotation = _currentRotation;
        _restartPosition = _trans.position;
        Cursor.lockState = CursorLockMode.Locked;

        if (showMarker)
        {
            _marker = GameObject.CreatePrimitive (MarkerPrimitive);
            _marker.name = MarkerObjectName;
            _marker.GetComponent<Renderer> ().sharedMaterial.color = markerColor;
            _marker.GetComponent<Collider> ().enabled = false;
            _markerTrans = _marker.transform;
        }

        renderCamera.targetTexture = new RenderTexture (TileWidth, TileHeight, 24);
    }

    private void Update ()
    {
        if (Cursor.lockState == CursorLockMode.None)
        {
            if (Input.GetButtonDown ("Fire1"))
            {
                Cursor.lockState = CursorLockMode.Locked;
            }

            return;
        }

        if (Input.GetKeyDown (KeyCode.Escape))
        {
            Cursor.lockState = CursorLockMode.None;
            return;
        }

        if (Input.GetButtonDown ("Fire2"))
        {
            _currentRotation = _restartRotation;
            _mainCameraTrans.eulerAngles = _currentRotation * rotationSpeed;
            _renderCameraTrans.eulerAngles = _currentRotation * rotationSpeed;
            _trans.position = _restartPosition;
            return;
        }

        Ray ray = mainCamera.ScreenPointToRay (Input.mousePosition);
        if (!Physics.Raycast (ray.origin, _mainCameraTrans.forward * mainCamera.farClipPlane, out RaycastHit hit))
        {
            if (showMarker)
            {
                _marker.SetActive (false);
            }
        }
        else
        {
            if (showMarker)
            {
                _marker.SetActive (true);
                float currentDistance = Vector3.Distance (_trans.position, hit.point);
                float adjustScale = Mathf.Sin (Time.time * MarkerScaleSpeed) * MarkerScaleAmount;
                float finalScale = currentDistance * adjustScale;
                _markerTrans.localScale = new Vector3 (finalScale, finalScale, finalScale);
                _markerTrans.position =
                    new Vector3 (hit.point.x, hit.point.y, hit.point.z) -
                    _mainCameraTrans.forward * (Mathf.Abs (finalScale) * MarkerPositionOffset);
                _markerTrans.Rotate (Vector3.up, Time.deltaTime * MarkerRotationSpeed);
            }

            if (Input.GetButtonDown ("Fire1"))
            {
                // Jutting out perpendicularly a little bit by the normal helps avoid clipping
                _trans.position = hit.point + hit.normal * CameraHeight;
                _mainCameraTrans.LookAt (hit.point);
                _renderCameraTrans.LookAt (hit.point);
                TeleportToPosition (_trans.position);
            }
        }

        if (Input.GetKeyDown (KeyCode.P))
        {
            _collider.enabled = !_collider.enabled;
            _rigidbody.isKinematic = !_rigidbody.isKinematic;
        }

        bool isPressingPageUpKey = Input.GetKey (KeyCode.PageUp) || Input.GetKey (KeyCode.E);
        bool isPressingPageDownKey = Input.GetKey (KeyCode.PageDown) || Input.GetKey (KeyCode.Q);
        bool isPressingRunKey = Input.GetKey (KeyCode.LeftShift) || Input.GetKey (KeyCode.RightShift);

        float movementSpeed = Time.deltaTime * (isPressingRunKey ? runSpeed : walkSpeed);
        float x = Input.GetAxisRaw ("Horizontal") * movementSpeed;
        float z = Input.GetAxisRaw ("Vertical") * movementSpeed;
        float y = isPressingPageUpKey ? movementSpeed : isPressingPageDownKey ? -movementSpeed : 0f;

        // Need to temporarily set rotation to the same as camera, so .forward is correct
        Quaternion tempRot = _trans.rotation;
        _trans.rotation = _mainCameraTrans.rotation;
        _trans.Translate (x, y, z, Space.Self);
        _trans.Translate (0f, y, 0f, Space.World);
        // ReSharper disable once Unity.InefficientPropertyAccess
        _trans.rotation = tempRot;

        _currentRotation.y += Input.GetAxis ("Mouse X");
        _currentRotation.x -= Input.GetAxis ("Mouse Y") * (invertYAxis ? -1f : 1f);
        _mainCameraTrans.eulerAngles = _currentRotation * rotationSpeed;
        _renderCameraTrans.eulerAngles = _currentRotation * rotationSpeed;
    }

    private void TeleportToPosition (Vector3 position)
    {
        if (_isWorkingOnMetadata || _isWorkingOnPanorama)
        {
            Debug.LogError ("Already working on another panorama.");
            return;
        }

        _loadChain = new LoadChain {location = landscape.origin.GetLocation (position.x, position.z)};

        lock (_lockObject)
        {
            _isWorkingOnMetadata = true;
            _metadataCallbackAction = CallbackFetchedMetadata;

            string url = "https://maps.googleapis.com/maps/api/streetview/metadata?" +
                         "key=" + streetViewApiKey +
                         "&size=" + TileWidth + "x" + TileHeight +
                         "&location=" + _loadChain.location.latitude.ToString (DoublePrecision) + "," +
                         _loadChain.location.longitude.ToString (DoublePrecision) +
                         "&heading=" + 0d +
                         "&pitch=" + 0d +
                         "&fov=" + Fov +
                         "&sensor=false";

            StartCoroutine (FetchJsonClass<PanoObject> (url, CallbackFetchedMetadataJson));
        }
    }

    private void CallbackFetchedMetadataJson (bool success, PanoObject panorama, string message)
    {
        if (!success)
        {
            _isWorkingOnMetadata = false;
            _loadChain.success = false;
            _loadChain.panorama = panorama;
            _loadChain.message = message;
            _metadataCallbackAction ();
            return;
        }

        if (panorama.status != "OK")
        {
            _isWorkingOnMetadata = false;
            _loadChain.success = false;
            _loadChain.panorama = panorama;
            _loadChain.message = panorama.status;
            _metadataCallbackAction ();
            return;
        }

        _isWorkingOnMetadata = false;
        _loadChain.panorama = panorama;
        _metadataCallbackAction ();
    }

    private void CallbackFetchedMetadata ()
    {
        if (!_loadChain.success)
        {
            onImageLoaded?.Invoke (_loadChain);
            return;
        }

        lock (_lockObject)
        {
            _isWorkingOnPanorama = true;
            Shader skyboxShader = Shader.Find ("Skybox/6 Sided");
            _loadChain.material = new Material (skyboxShader);
            _imageCallbackAction = CallbackFetchedStreetView;
            _timeoutTimer = Time.time + WaitTimeout;
            _fetchesDone = 0;

            RequestPanoramaImage (_loadChain.location, 0d, 0d, CallbackPanoramaImageFront);
            RequestPanoramaImage (_loadChain.location, 90d, 0d, CallbackPanoramaImageLeft);
            RequestPanoramaImage (_loadChain.location, 180d, 0d, CallbackPanoramaImageBack);
            RequestPanoramaImage (_loadChain.location, 270d, 0d, CallbackPanoramaImageRight);
            RequestPanoramaImage (_loadChain.location, 0d, 90d, CallbackPanoramaImageUp);
            RequestPanoramaImage (_loadChain.location, 0d, -90d, CallbackPanoramaImageDown);
        }
    }

    private void RequestPanoramaImage (
        GeoCoord location, double heading, double pitch, Action<bool, Texture2D, string> callback)
    {
        string url = "https://maps.googleapis.com/maps/api/streetview?" +
                     "key=" + streetViewApiKey +
                     "&size=" + TileWidth + "x" + TileHeight +
                     "&location=" + location.latitude + "," + location.longitude +
                     "&heading=" + heading +
                     "&pitch=" + pitch +
                     "&fov=" + Fov +
                     "&sensor=false";

        StartCoroutine (FetchImage (url, callback));
    }

    private void CallbackPanoramaImage (int side, bool success, Texture result, string message)
    {
        lock (_lockObject)
        {
            if (!success)
            {
                _isWorkingOnPanorama = false;
                _loadChain.success = false;
                _loadChain.message = message;
                return;
            }

            result.wrapMode = TextureWrapMode.Clamp;
            _loadChain.material.SetTexture (side, result);
            _fetchesDone++;

            if (Time.time > _timeoutTimer)
            {
                _isWorkingOnPanorama = false;
                _loadChain.success = false;
                _loadChain.message = "Attempt to fetch StreetView panorama timed out";
            }

            if (!_loadChain.success)
            {
                _imageCallbackAction ();
                return;
            }

            if (_fetchesDone < NumberSides)
            {
                return;
            }

            _imageCallbackAction ();
            _isWorkingOnPanorama = false;
        }
    }

    private void CallbackPanoramaImageFront (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_frontTex, success, texture, message);
    }

    private void CallbackPanoramaImageLeft (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_leftTex, success, texture, message);
    }

    private void CallbackPanoramaImageBack (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_backTex, success, texture, message);
    }

    private void CallbackPanoramaImageRight (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_rightTex, success, texture, message);
    }

    private void CallbackPanoramaImageUp (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_upTex, success, texture, message);
    }

    private void CallbackPanoramaImageDown (bool success, Texture2D texture, string message)
    {
        CallbackPanoramaImage (_downTex, success, texture, message);
    }

    private void CallbackFetchedStreetView ()
    {
        if (!_loadChain.success)
        {
            onImageLoaded?.Invoke (_loadChain);
            return;
        }

        RenderSettings.skybox = _loadChain.material;

        GeoCoord revisedLocation = new GeoCoord (_loadChain.panorama.location.lat, _loadChain.panorama.location.lng);
        Vector3 revisedPosition = revisedLocation.GetMapCoord (landscape.origin).GetPosition ();
        Vector3 pointAboveTerrain = revisedPosition;
        pointAboveTerrain.y -= 5000;
        Vector3 pointBelowTerrain = new Vector3 (pointAboveTerrain.x, pointAboveTerrain.y + 10000, pointAboveTerrain.z);
        Vector3 rayDirection = pointAboveTerrain - pointBelowTerrain;
        Ray ray = new Ray (pointBelowTerrain, rayDirection);
        bool didHitTerrain = Physics.Raycast (ray, out RaycastHit hit, Mathf.Infinity);
        if (!didHitTerrain)
        {
            return;
        }

        revisedPosition = hit.point;
        revisedPosition.y += CameraHeight;
        _trans.position = revisedPosition;

        _loadChain.success = true;
        _loadChain.message = revisedLocation.ToString ();
        _loadChain.renderTexture = renderCamera.targetTexture;
        onImageLoaded?.Invoke (_loadChain);
    }

    private static IEnumerator FetchJsonClass<T> (string url, Action<bool, T, string> callback)
    {
        bool success = true;
        string message = null;

        UnityWebRequest www = UnityWebRequest.Get (url);
        yield return www.SendWebRequest ();

        if (www.isHttpError || www.isNetworkError || !string.IsNullOrEmpty (www.error))
        {
            success = false;
            message = "Network error " + www.error + " attempting to fetch URL " + url;
        }

        T result = JsonUtility.FromJson<T> (www.downloadHandler.text);
        if (result == null)
        {
            success = false;
            message = "No JSON data could be parsed from URL " + url;
        }

        callback?.Invoke (success, result, message);
    }

    private static IEnumerator FetchImage (string url, Action<bool, Texture2D, string> callback)
    {
        bool success = true;
        string message = null;

        UnityWebRequest www = UnityWebRequestTexture.GetTexture (url);
        yield return www.SendWebRequest ();

        if (www.isHttpError || www.isNetworkError || !string.IsNullOrEmpty (www.error))
        {
            success = false;
            message = "Network error " + www.error + " attempting to fetch URL " + url;
        }

        Texture2D result = ((DownloadHandlerTexture) www.downloadHandler).texture;
        if (result == null)
        {
            success = false;
            message = "No texture data could be parsed from URL " + url;
        }

        callback?.Invoke (success, result, message);
    }

    // These names are required to be in these formats by the JSON conversion and can't be changed
    // ReSharper disable IdentifierTypo
    // ReSharper disable StringLiteralTypo
    // ReSharper disable NotAccessedField.Global
    // ReSharper disable InconsistentNaming
    [Serializable]
    public class PanoObject
    {
        public string copyright;
        public string date;
        public Location location;
        public string pano_id;
        public string status;
    }
    // ReSharper restore InconsistentNaming
    // ReSharper restore NotAccessedField.Global
    // ReSharper restore StringLiteralTypo
    // ReSharper restore IdentifierTypo

    // These names are required to be in these formats by the JSON conversion and can't be changed
    [Serializable]
    public class Location
    {
        public double lat;
        public double lng;

        public override string ToString ()
        {
            return lat + "," + lng;
        }
    }

    [Serializable]
    public class LoadChain
    {
        public bool success;
        public GeoCoord location;
        public PanoObject panorama;
        public RenderTexture renderTexture;
        public Material material;
        public string message;

        public LoadChain ()
        {
            success = true;
        }
    }
}